Freenet Explained
[200] Part 2: For developers
[200] Part 2: For developers
Something about metadata: Every key in Freenet has metadata attached (maybe empty one though). So when you want to write freenet tools, you have to know how metadata looks like.
Metadata
Metadata is something that Freenet inserts along with a file, which allows Freenet clients to know something about it. Usually, it will include the filetype, but also can contain some very important information like what sort of SplitFile chunks there are or can even redirect the client to another Freenet file.
To see metadata for a key, type java -cp <freenet directory>/freenet.jar freenet.client.cli.Main get --noredirect <key> <file> and the metadata from the file will be displayed on the screen.
DBR's
About
Since content in Freenet cannot be updated, authoring a freesite can be difficult. To make it easier, there is a type of redirect that automatically figures out the date and redirects you to the latest version of the file if it exists. Thus, by redirecting users to a new file with the date as part of the key, a freesite author can insert new versions of the site periodically and have the DBR automatically point to the right version. Usually a DBR is set for 24 hours, but this is not a requirement. Most popular freesites today use 24-hour DBR's to transparently simulate updating. This is why many sites change at midnight GMT.
If a DBR existed at
freenet:SSK@abc/foo/bar, then you might access content there by typing in
freenet:SSK@abc/foo/bar//index.html. The DBR will actually point to something like
freenet:SSK@abc/foo/3c057a80-bar, which will then contain information on how to access index.html. The 3c057a80- is the number of seconds since epoch (Jan. 1, 1970) in hex. This is all invisible to end users.
An easy way to retrieve older versions of DBR pages, when using your browser, is to append ?date=YYYYMMDD at the end of the URI for the site.
Howto
To download a DBR, you don't have to do anything. All tools and fproxy will handle this automatically. To insert one for a 24 hour period, enter the command fcpputsite <name> <directory of files to insert> <ssk public key> <ssk private key> which inserts the DBR at
freenet:SSK@<ssk public key>/<name>. Alternatively, you can use the command java -cp <freenet directory>/freenet.jar freenet.client.cli.Main putsite <URI> <files>, where <URI> would be for example
freenet:SSK@<private key>/<name>.
Format
A DBR has the following form:
Version
Revision=1
EndPart
Document
DateRedirect.Target=freenet:SSK@npfV5XQijFkF6sXZvuO0o~kG4wEPAgM/04_homepage
End
Map Files
About
Map files, also called MSK's or manifests, are blank files with metadata that points to various other files' CHK's. These are a mainstay of current freesites because they allow only one file to redirect to many files, decreasing the amount of lookups fproxy needs to do. If you see <map>//<file> in a URI, that tells fproxy to look at the metadata of <map> and find the CHK for <file>.
Format
Map files follow the form:
Version
Revision=1
EndPart
Document
Redirect.Target=freenet:CHK@Yr45pILWZ5JG0SxO56BXdM22CnoLAwI,FVLEhvvQxRwNckgnOWOYvQ
Name=history.txt
Info.Format=text/plain
EndPart
Document
Redirect.Target=freenet:CHK@BSI9FMrG7YpKrZXX3jm-- nF6mesLAwI,7aKDjQmNNqbwIP9ah~l0fg
Name=nube.gpg.txt
Info.Format=text/plain
End
About
SplitFiles are a pretty useful concept. The idea is that one file is broken up into many smaller chunks. All the chunks are inserted, as well as one file that contains metadata that points to all these chunks (this is the file that you insert the SplitFile under). This improves reliability of the network by increasing redundancy. Different chunks can be gotten from different computers, it is harder to discern which files are being transfered, and the possibility of chunk collisions is increased (a good thing). Also, this allows one to resume a download easier by merely downloading the SplitFiles you haven't yet. All in all, a very sexy idea.
Howto
Right now, the easiest way (IMO) to insert and retrieve SplitFiles is through fproxy with FEC (see below), but if you don't want to bother setting that up, you can use the FCP tools provided with snapshots of Freenet (should be under freenet/clients). To get the newest version, check it out from Freenet CVS under Contrib/fcptools.
There is an executable, called fcpputsplit, for which I recommend you use these parameters: fcpputsplit KEY FILE. KEY is the freenet key you want your new file to have. FILE is the filename on your harddrive. The default size is 256 kb for each SplitFile.
Encoding/Decoding SplitFiles can be pretty hard on your CPU. Expect 100% usage for a bit.
Format
The format of SplitFiles follows the form:
Version
Revision=1
EndPart
Document
Info.Format=application/x-gtar
SplitFile.Size=12b14b
SplitFile.Block.5=freenet:CHK@V~eOY-QVTDexy5Yyt9Xpyf-dwF8SAwI,KMUx3JU2o7dvXJkvTvAm8w
SplitFile.BlockCount=5
SplitFile.Block.4=freenet:CHK@kJn-v7nH5O2bb7Xx-l7RA5k3sbESAwI,Tcl 6x B3 Nzl Y0p 0 L?-Pp Y1u Q?
SplitFile.Block.3=freenet:CHK@yH53sGikKBalUTRfk6JvX778N6ISAwI,Af O Lj QY9b Ig JC0 Yf Pf L1 Rw?
SplitFile.Block.2=freenet:CHK@Tnx1nIXTF5HRHSC90BPs~T1OWkwSAwI,gCh7sPS4B1S7NozRZMtUtg
SplitFile.Block.1=freenet:CHK@OK7qw-vtBvPIKgcKzWIeU2iY7HYSAwI,B Eo J9 YP Ux ZC2y Lcc G3h 20g?
End
Forward Error Correction
About
Since missing one SplitFile chunk makes the whole download useless, forward error correction was added to fproxy by gj. FEC creates check files when inserting and inserts these check files as well as the normal SplitFiles. A check file, when combined with the FEC decoder and some of the SplitFiles, can create other, missing SplitFiles. Thus, only a certain percentage of the SplitFiles/check blocks need to be downloading. This allows for the unstable nature of Freenet not to fuck you over when a few of the blocks are missing.
Howto
The FEC support in fproxy is done using some code from Onion Networks. It should be enabled by default. It works transparently when you upload KSKs or . Since it is currently tied to fproxy, you must use the Freenet Gateway (usually at localhost:8888) to insert and retrieve files with FEC. It will automatically choose the SplitFile size when inserting (with the hopes that standard SplitFile sizes will increase the chance of two SplitFiles will be the same) and provides a neat little status page.
Format
The format of FEC metadata looks a lot like normal SplitFiles:
Version
Revision=1
EndPart
Document
Info.Format=application/x-gtar
Info.Description=Redundant SplitFile created by FProxy
SplitFile.decoder.name=OnionDecoder_0
SplitFile.decoder.segments=1
SplitFile.decoder.baseN=7
SplitFile.decoder.encoder=OnionEncoder_0
SplitFile.decoder.endN=7
SplitFile.decoder.stripeWidth=-1
SplitFile.decoder.baseK=5
SplitFile.decoder.endK=5
SplitFile.Block.5=freenet:CHK@bZ4bmhji0bXfPPQNB0wqjLfACEASAwI,7S Jf Ooys 0qdx D Ymiy RG Im A?
SplitFile.Size=12b14b
SplitFile.Block.4=freenet:CHK@kJn-v7nH5O2bb7Xx-l7RA5k3sbESAwI,Tcl 6x B3 Nzl Y0p 0 L?-Pp Y1u Q?
SplitFile.Block.3=freenet:CHK@yH53sGikKBalUTRfk6JvX778N6ISAwI,Af O Lj QY9b Ig JC0 Yf Pf L1 Rw?
SplitFile.Block.2=freenet:CHK@Tnx 1n IXTF5 HRHSC90 B Ps?~T1OWkwSAwI,gCh7sPS4B1S7NozRZMtUtg
SplitFile.Block.1=freenet:CHK@OK7qw-vtBvPIKgcKzWIeU2iY7HYSAwI,B Eo J9 YP Ux ZC2y Lcc G3h 20g?
SplitFile.BlockCount=5
SplitFile.BlockSize=40000
SplitFile.CheckBlock.2=freenet:CHK@wYYilpDhlLe9WNMs7J7GBDuNB1gSAwI,VB~wMEzYw1FUKpf2Eq~eKw
SplitFile.CheckBlock.1=freenet:CHK@a0vap2H0UAj9BNa1Q7gfTK3XjFISAwI,ousM-feXPLSPqZdc5aq3NQ
SplitFile.CheckBlockCount=2
End
Native FEC
If you are interested in getting Native FEC -- some much faster OS-specific code as opposed to the Java code used by default -- then follow these steps:
Extract the zip. If you are using Linux, put fec-linux-x86.jar in your freenet directory. If you are using Windows, put fec-win32.jar in your freenet directory.
Change the following lines in freenet.conf:
%FEC.Decoders.0.class=Onion FEC Decoder
? to FEC.Decoders.0.class=Native Onion FEC Decoder
?
%FEC.Encoders.0.class=Onion FEC Encoder
? to FEC.Encoders.0.class=Native Onion FEC Encoder
?
Note very carefully the removal of the '%' character. If you do not remove this, your change will not be noticed by Freenet.
Now, we change the Freenet loader to know about our fec-*.jar. If you're use Linux, open up start-freenet.sh for editing and change CLASSPATH=freenet.jar:freenet-ext.jar:$CLASSPATH to CLASSPATH=fec-linux-x86.jar:freenet.jar:freenet-ext.jar:$CLASSPATH. If you're running Windows, please let me know how to change the classpath, since I don't know.
BAM! You're done. Next time you load Freenet, you'll have a sexier, faster algorithm for FEC encoding/decoding. Wasn't that worth it?
Generally, metadata looks like
Header1
Name1=Value1
Name2=Value2
...
EndPart
Header2
Name1=Value1
...
EndPart
Header2
Name1=Value1
...
End
The order of the headers and of the values within the headers does not matter at all. One header may appear multiple times with different attributes. All data after the final End (and the newline after it) are discarded. Usually, non-empty metadata contains a Version block with Revision=1 as first header. All the other headers are usally Document headers. The Version block can contain Encoding=gzip as well, which means that the rest of the metadata is gzipped, but I have never seen that "in the wild". All numbers are in hex.
Name: Example: Usage:
empty metadata
for SplitFile segments or sometimes for files redirected to by a redirect or a mapfile.
Minimal metadata
Version
Revision=1
End
files redirected to by a redirect or mapfile.
Content type/charset only
Version
Revision=1
EndPart
Document
Info.Format=text/html;charset=utf-8
End
For "standalone" files and for containers. in case of text/html you can specify a character set that should be used by the anonymity filter and will be reported to the browser.
Redirect
Version
Revision=1
EndPart
Document
Info.Format=text/html
Redirect.Target=freenet:KSK@gpl.txt
End
Used for "standalone files" that redirect to a key with minimal or no metadata. Usually the target is a CHK key. The content type might be omitted here and instead placed in the metadata of the target. In case of a container, the content type application/zip may only be mentioned in the metadata of the container file itself, not in any redirects.
DateRedirect
Version
Revision=1
EndPart
Document
Info.Format=text/html
DateRedirect.Increment=15180
DateRedirect.Offset=0
DateRedirect.Target=freenet:KSK@gpl.txt
End
Used for DBR sites - usually redirects to a SSK key with a manifest in it, but KSK is also possible (but hijackable). The increment and offset can be left out, if they are default values (as shown in this example, all values in seconds.)
Manifest file/mapfile
Version
Revision=1
EndPart
Document
Name=index.html
Info.Format=text/html
Redirect.Target=freenet:CHK@foo
EndPart
Document
Name=image.png
Info.Format=text/html
Redirect.Target=freenet:CHK@bar
EndPart
Document
Name=image.png
Info.Format=text/html
Redirect.Target=freenet:CHK@foo
End
Used for freesites in general. To access a site in here, add //name to the end of the freenet key. If only // is added, the default key (the one without Name) is fetched. The order of the attributes within a section can be used to "guess" the insertion tool that inserted this site.
SplitFile
Version
Revision=1
EndPart
Document
Info.Description=file
Info.Format=application/zip
Info.Checksum=15a281c3382529a50eafe526cdb0ceae60012e21
SplitFile.AlgoName=OnionFEC_a_1_2
SplitFile.Size=8ebd2
SplitFile.BlockCount=6
SplitFile.CheckBlockCount=3
SplitFile.Block.1=freenet:CHK@...
SplitFile.Block.2=freenet:CHK@...
SplitFile.Block.3=freenet:CHK@...
SplitFile.Block.4=freenet:CHK@...
SplitFile.Block.5=freenet:CHK@...
SplitFile.Block.6=freenet:CHK@...
SplitFile.CheckBlock.3=freenet:CHK@...
SplitFile.CheckBlock.2=freenet:CHK@...
SplitFile.CheckBlock.1=freenet:CHK@...
End
Used for FEC SplitFiles. The checksum is SHA1, all numbers (the block and check block numbers as well) are in hex as usual.
Fproxy
About
Fproxy is a layer between your Freenet node and your browser that makes it easy to access Freenet content. It comes with Freenet and loads by default. Just point your browser to localhost:8888 and the Freenet Gateway page should open up. This is the starting point for fproxy.
Howto
To access data through fproxy, you must specify the key you want to access after the
http://localhost:8888/. For example, to access the key KSK@gpl.txt, you could type in
http://localhost:8888/KSK@gpl.txt. Alternatively, you can enter the key in the appropriate textbox on the Freenet Gateway.
To insert data through frpoxy, you must fill in the form on the Freenet Gateway and fproxy will do the rest. It will automatically insert it as a SplitFile and use FEC encoding. While it is inserting, you are presented with a nice status screen telling you how far it's gotten.
Configuration
Fproxy is a part of the default installation. To configure it, you must edit the freenet.conf/freenet.ini file in your Freenet directory. For documentation on all the various options, read the automatically generated command line documentation.
FCPTools
About
FCPTools is actually a collection of useful command line tools. It contains fcpget, fcpproxy, fcpput, fcpputsite, and fcpputsplit.
Downloading
Here's the easiest way to get the tool set:
For Linux:
tar xvfz fcptools-stable.tgz
cd fcptools-stable
./configure
make
make install #as root
For Windows:
Unzip and enjoy.
For Mac:
Check out the cvs tree, compile. Sorry, I don't know more about the Mac tools for this.
Howto
Honestly, these are pretty self-explanatory. Since the programs are currently in heavy development, your best bet is to run <program> --help.
GJ's Manifest Tools
About
GJ's Manifest Tools, accessible from GJ's site, allow one to pull all files from a map file and put them in a directory (Get Files
?) or put all files in a directory into Freenet and uploads a map file listing them (Put Files
?).
Howto
Download or compile the manifest.jar from GJ's site. To run Put Files
? or Get Files
?, type the following commands:
java -cp manifest.jar:freenet.jar relentless.gj.cli.Put Files
?
java -cp manifest.jar:freenet.jar relentless.gj.cli.Get Files
?
Where freenet.jar is your jar file for Freenet itself (kept in your freenet directory under libs). Running these commands without any arguments gives usage information.
Freenet Message Board
About
FMB is a client designed for text messaging over Freenet. You can find a freenet-based site for it here.
The basic idea is simple: You create an identity. This identity has an SSK associated with it (generated by the program or provided by you). When you write a message and insert it, you are inserting it under your SSK. You choose the identities you want to receive messages from, and their messages will then be downloaded from their SSK. In addition, messages they have downloaded will be sent to you, but marked as unverified. This is because they could have inserted a fake message, purporting to be from some other identity. To verify the message, you can just look at the other person's SSK.
When you subscribe to someone, you download all the messages they write -- not only ones to you. There is no concept of a private message. Thus, if you want secure communication with someone, use GPG.
Currently (up to at least alpha4a), FMB only polls for messages written the same day you are requesting them, and ignores the GMT day switch if you are running it while that occurs. Thus, please restart FMB after GMT switches over.
Howto
This was all written with version alpha4a in mind. If you have a different one, take this all with a grain of salt.
Main Screen: You will see a message viewer on the left. This shows messages you know about. Below that is a message reader, displaying the currently selected message. If you got this message from a third party, you can click on the red verify button to make sure it is valid. If the message is verified, the verify button will be greyed out. On the right hand side is a listing of identities you know about.
Requesting Messages: To start reading some messages, right click on the identity you want to receive from. Pick start receiving messages. Now, over time, you will see all the messages they wrote, knew about. The progress bar under their name will fill with X's, R's, I's, numbers, and ?'s. X=success, R=receiving, I=inserting, numbers=retries, ?=invalid data.
Writing Messages: To write a message, click on the big create a new message button. If you want to reply, instead click on the reply to this message button under the message reader window. When it is inserted into your SSK, other people can then read it by subscribing to you (or getting it through a third party). When you write messages, they go into your buffer. You must manually insert your buffer by right clicking on your nick and choosing send buffered inserts now. This will insert all the messages you wrote together.
Identity Comments: Each identity has a comment board associated with it. To view comments on an identity, right click their name and select show information about this user. This will show you some info they entered about themselves as well as Good/Neutral/Evil assignments from other identities. You can use these comments as a basis for listening to an identity or not. It is considered polite to leave Neutral comments about someone just to let them know you got their identity anouncement.
Archives: Since messages are only downloaded for the current day, you might like to have access to old messages or current messages if you can't seem to download them. Thus, you can download one of the message archives provided by other identities. Click on the archives tab and select an archive from an identity you trust and that has messages that are new to you (displayed under the new column). Left click on the archive to select it, then right click on the archive, select request archive, and wait. If you can retrieve the archive, all their messages are added to your message pool. If you feel like you have a comprehensive message pool or have messages that other people missed for some reason, you can insert your own archive by clicking the insert message archive button.
Chess: Another neat feature of FMB is anonymous chess. Click on the chess lounge tab to view current games. There is a drop-down list of current people willing to play. To play them, select their name and click play game. To be put on the list of people waiting for opponents, click add me to that list. You can view a currently progressing game by first left clicking on it to select the game, and then right clicking on it and selecting show this game.
[205] What classes are responsible for the Next Generation Routing function?
In terms of actual routing...freenet/node/rt/ has all the code.
Routing Table
? is the general interface.
CPAlgoRoutingTable is the classic routing table.
NGRoutingTable is the NGR routing table.
Routing is used to represent a list of routes, returned from the Routing Table
?. NGRouting is the NGR implementation of this.
NGRoutingTable keeps a hashtable of nodes and a Standard Node Estimator
? for each, which keeps running averages and estimators for that node. For example, the probability of a query being rejected (a running average), or the probability of a request producing a DNF (a Key Estimator
?).
Running Average
?'s are used to track any variable that is "flat" i.e. not
dependant on the key being routed. pQR is an example as above.
There are several implementations:
Simple Running Average
? - average of last N reports.
Simple Intervalled Running Average
? - average of reports over last specified time period
Time Decaying Running Average
? - time biased (by a half-life) average of all
reports ever (uses some mathematical tricks so doesn't need to store them).
Simple Binary Running Average
? - average of last N reports specialized for
probability estimators, so each report is either 0.0 (no event), or 1.0
(event happened). So it only needs to keep 1 bit per report.
Decaying Running Average
? - kalman filter, essentially. curValue ->
decayFactor * report + (1-decayFactor) * curValue. We arbitrarily use a
decayFactor of 0.2 at this point; there is room for some research in
this area.
Self Adjusting Decaying Running Average
? - adaptation of DRA for probabilities, no longer used due to gross inaccuracy.
Key Estimator
?'s are used to track any variable that is dependant on the key being routed. They track a distribution across the keyspace, usually dividing it into a fixed number of points or sectors (of variable size).
Currently Key Estimator
? has one main implementation:
Decaying Keyspace Estimator
?. This is the "gravity" algorithm described on ian's paper. It is not very accurate, I am writing a new implementation based on a different principle, Sliding Buckets Keyspace Estimator
?, which should be much less alchemical and more accurate.
However if you are talking about the forced specialization via pcaching
paper, you will need to look at the probabilistic caching system rather
than routing. You can start on that via freenet.node.Node.shouldCache().
Now, a warning: we are not likely to accept any patches which implement
a fixed, forced specialization, for various reasons. If you want
clarification on that, raise the point on devl. This means if you
implement a forced-specialization system, you will be essentially on
your own; you can test it with your own networks, or you can try to get
people to run nodes with such code (and many of them may), but you will
probably have to maintain your own branch.
Finally: there is much work to be done on NG Routing, and any help you
can give us would be of interest. Much of it may well be of academic
interest too.
[209] Minimum Request Interval (see
MRI)