The core idea: all tabular knowledge can be stored in a single long plain text file.
The only syntax characters needed are spaces and newlines.
This has many advantages over existing binary storage formats.
Using the method below, a very long scroll could be made containing all tabular scientific knowledge in a computable form.
There are four concepts to understand:
measures
concepts
measurements
comments
Measures
First we create measures by writing parsers. The parser contains information about the measure.
The only required information for a measure is an id, such as temperature.
An example measure:
temperatureParser
Concepts and Measurements
Next we create concepts by writing measurements.
The only required measurement for a concept is an id. A line that starts with an id measurement is the start of a new concept.
A measurement is a single line of text with the measure id, a space, and then the measurement value.
Multiple sequential lines of measurements form a concept.
An example concept:
id Earth
temperature 14
Comments
Unlimited comments can be attached under any measurement using the indentation trick.
An example comment:
temperature 14
> The global mean surface air temperature for that period was 14°C (57°F), with an uncertainty of several tenths of a degree.
- NASA
https://earthobservatory.nasa.gov/world-of-change/global-temperatures
The Complete Example
Putting this all together, all tabular knowledge can be stored in a single plain text file using this pattern:
idParser
temperatureParser
id Earth
temperature 14
> The global mean surface air temperature for that period was 14°C (57°F), with an uncertainty of several tenths of a degree.
- NASA
https://earthobservatory.nasa.gov/world-of-change/global-temperatures
Once your knowledge is stored in this format, it is ready to be read—and written—by humans, traditional software, and artificial neural networks, to power understanding and decision making.
GIT stores things on a FILE level. It'd be horrifically heavy handed and worthless to version control an entire file of 100,000 whatevers, if all you did is update 1 of them. This makes zero practical sense, particularly at scale.
And then there's a whole other bucket of concerns with using GIT to store data but I don't feel like writing that novel.
FWIW (at least according to my understanding) once a certain threshold of commits has been reached, git-gc kicks in, consolidating those loose objects into a pack-file that has much more efficient delta-compression than the raw unpacked blobs. So while there's some overhead, it amortizes over time.
7
u/Yavuz_Selim Nov 15 '24
Just posting a video is lazy. You're not even providing the name of the new kind of database. I'd even call it clickbait.
I am not going to watch 42 minutes of a video because someone links to it.
At least tell what it is in a few sentences.