Assume you have binary data files of size > 1GB.
These files contains events each marked by 585
in hex.
I convert the binary to hex by xxd -ps data.raw | sed ':a;N;$!ba;s/n//g'
.
I can get wanted hex position to binary back by |xxd -r
.
I characterize characteristics of data in hex about which I propose to create a table called hex
but I should have pointers into the binary data directly, since it is too expensive to convert everything to hex – back and forth.
I skip the table hex here.
I think you can get a list of pointers to the start position of each event.
I think the pointer could be stored as VARCHAR but I am not sure if it is the optimal one.
Table
CREATE TABLE events
(
event_id SERIAL PRIMARY KEY NOT NULL,
measurement_id INTEGER NOT NULL,
eventPointer VARCHAR NOT NULL,
is_active BOOLEAN DEFAULT FALSE
);
CREATE UNIQUE INDEX dir_events
ON events (measurement_id)
USING btree
(eventBody)
WHERE is_active;
CREATE OR REPLACE VIEW events_current AS
SELECT event_id, eventPointer,
FROM events
WHERE is_active
where the second entry creates the partial index, while the last one is a view to access the index always through the same function.
I think I need event_ID also but I am not sure if you can use pointer as an ID too. I think not.
How can you manage pointers well in partial index?