In case anyone's wondering, no I haven't died, and yes I'm still busy on this. I'd like to have something more-or-less usable up by the end of next week (albeit without backwards compatibility yet, most likely).
0MQ by Tewinget
dEBRUYNE posted this 8 years ago
XMR5,000.00
funded of XMR5,000.00 target
133 XMR of the remaining milestone has been sent to vtnerd for his work on completing this project; the remaining 867 will be delivered to the General Fund
First and foremost, this is more or less a formal proposal, since there is still about 5K XMR “stuck” in the forum funding system. This is due to another developer not delivering. The other proposal can be seen here:
https://forum.getmonero.org/9/work-in-progress/2355/improved-nvidia-mining-software
It also states:
“FYI - This a non-refundable system. If, for whatever reason, things don't work out, the funds are kept for future developers to receive.”
Therefore, we thought it would be appropriate to direct these funds to tewinget to work on 0MQ. However, I have created this proposal to get some input from the community and also to see if no one (or at least no majority), and in particular the funders of the other proposal, is opposed to it.
WHAT
Predominantly 0MQ, which is going to replace our interprocess RPC with IPC using 0MQ. Basically the first priority is to replace the RPC communication between the daemon and client apps (simplewallet, blockchain_* tools) and also create a stub application (monero-rpc-deprecated) to still be able to serve the calls that used to be in the daemon. Furthermore, all new code will be documented and new tests will be written.
WHO
I am writing this on behalf of Thomas Winget (tewinget), who has been contributing to the Monero codebase since approximately June 2014. His largest contribution is the migration from storing the blockchain in a static, binary format to storing the blockchain using a database API, as well as two BlockchainDB implementations (LMDB and BerkeleyDB). In addition, he documented and cleaned up the source code. Other contributions of his can be found via the network graph on Github or simply looking at his fork.
As a side effect of the work he has done already on Monero, he is rather familiar with most of the codebase. This puts him in a good position to work on 0MQ. Furthermore, he will further cement the knowledge he has of the codebase and gain knowledge in areas he is less familiar with, enabling him to efficiently implement new features in the future and guide others who may have questions regarding a specific module.
WHY
First and foremost, 0MQ is way more efficient than the current RPC. In addition, 0MQ will enable walletnotify / blocknotify functionality. It will also enable a new RPC API implementation and a Bitcoin compatible RPC API implementation. The latter will arguably make it easier for merchants / exchanges / wallet services to implement Monero.
PROPOSAL AND MILESTONES
At first, tewinget is going to spend approximately 10 hours per week working on 0MQ and if possible more. In addition, the hours might increase in the future. His rate currently is 25 XMR, which is approximately 22$, taking the 30 day weighted average dollar price from:
http://moneroprice.i2p.xyz/
NOTE: This proposal was actually written approximately 2-3 weeks ago and tewinget also basically started to work on 0MQ then. However, Fluffypony was travelling and was therefore unfortunately unable to review the proposal. Back then, we were bouncing around 85-90cents. Bear in mind that the developer also takes a huge risk by taking payment in XMR. For all we know, the price would have slided down further, like the last time (i.e. autumn / winter) Bitcoin went up significantly. I genuinely hope the community doesn't have any issues with these, because 0MQ is a critical part of our ecosystem.
tewinget will first start working on 0MQ and provide some more concrete milestones (with the hours it is going to take) further on. In addition, notes will be kept which will show how many hours was spent on everything. For now, payouts will be made monthly corresponding to the hours worked.
You can follow his activity here:
https://github.com/tewinget?tab=activity https://github.com/tewinget/bitmonero/tree/zmq-dev
As you can see he is already actively working on 0MQ
P.S. tewinget also briefly discussed in the dev meeting yesterday how things were going. I hope to push out the logs tonight so you can all read that as well.
Update: 40 more hours complete, daemon RPC (mostly) complete, wallet now coded to use new RPC. PR submitted, but needs polish before merging. At this time I don't think there are any non-starter issues though, so testing should be an option.
I'll update here again when I think the PR is in a merge-able state.
Okay, so...it's been quite some time (and well more than 40 hours of work) since my last update, but at things are now in a state where I think they're ready to be merged. Or at least reviewed for merge (meaning others will likely suggest tweaks/changes before merging, and I'll oblige after discussion). The merge is done as of a couple days ago, and from here I'll keep it merged with upstream as well as possible until it's merged into upstream.
Changes since last update:
basically lots of (mostly) little bug fixes.
Well, also there's that little thing called RingCT...I managed to leave out RingCT-related things as of the last update, but that's in now.
Things I'll be working on while monitoring feedback and doing some more testing:
making sure I haven't inadvertently messed up some functionality in the transition
documenting the new RPC format and any changes (along with justification for those changes)
Next on the list after merge:
- You guys have been waiting long enough for zmq -- my apologies -- and so far it's just a (better, I hope!) re-implementation of existing things. As such, I think next after merge I'll tackle {block,transaction}-notify.
Oh, and now that I think about it (the thought came to me just now, as I'm typing this), I'll probably need to deal with moving zmq source into our project (as with libunbound, lmdb (I think?) and others), as I believe the plan is to not have it as an external dependency.
Ok, time for an update! A majority of the daemon RPC commands are now complete:
GetHeight
GetBlocksFast
GetHashesFast
GetTransactions
KeyImagesSpent
GetTxGlobalOutputIndices
GetRandomOutputsForAmounts
SendRawTx
GetInfo
SaveBC
GetBlockHash
GetLastBlockHeader
GetBlockHeaderByHash
GetBlockHeaderByHeight
GetPeerList
SetLogLevel
GetTransactionPool
HardForkInfo
GetOutputHistogram
GetRPCVersion
At least half of them have been tested (some are easier to test than others, e.g. I'll test SendRawTx on testnet with the wallet software rather than with my python script).
In addition, the code for cryptonote objects to/from json has been improved (yay, can handle pretty much infinitely-nested containers now!).
A basic libdaemonrpc (daemon_rpc_client) is in place, and the wallet uses it for GetHeight, but that's not yet tested.
As always, for a complete overview of changes, see the branch on my github.
The python script I've been using to test the RPC commands can be found here. If it were more of a long-term thing I'd put it up somewhere on github, but for now it's just local, hence the paste.
If anyone wants to test some things in the near future, I'll probably make a comment here with instructions, but also feel free to chat with me in #monero-dev on freenode.
At just over 80 hours now (we'll call it 80), a decent amount of that time has been spent planning/designing. Since a large amount of the design considerations are taken care of, look for much more progress in the coming weeks!
Progress so far (after ~40 hours):
In order to facilitate easier and simpler development of alternate-language implementations using the Monero RPC via 0MQ, it makes sense to me to package all RPC commands (and all relevant C++ classes) as json objects. This will result in a bit slower code, sure, but that slowdown will be insignificant and I believe the benefits heavily outweigh that cost.
To that end, I've written serialization to/from json for a large number of the C++ classes (more to come as needed). With coding, testing, fighting the compiler, sacrificing goats, etc. it's been about 18 hours of work.
The remaining ~20 hours (of the ~40 so far) have yielded three fully implemented RPC commands working via 0MQ. As expected, the first command and the 0MQ integration took up a bulk of that time, and as such more RPC commands should follow quickly.
Some of the code is tentative, and some of it lacks proper error handling (as I don't want to write in error handling only to figure out I've screwed something up and have to redo the implementation and the error handling).
Anyone looking to test out a current build will need libzmq somewhere that CMake can find it, as I've not yet dealt with pulling zmq's source into our external libs compilation.
Any feedback is welcome and encouraged, either here, on IRC, or via my github.
As the organizer of the failed nvidia proposal, and a small contributor to the proposal, I guess I should say something.
In general, I support this measure to move these funds to something that will strengthen the codebase and "innerworkings" of Monero. However, I will raise the point that hyc did, indeed, provide solo mining support for the nvidia miner. This was part F of the original proposal,
> F) Works as a solo miner - is able to communicate directly with the monero wallet software. EDITED - Solo mining requires working with the daemon. Sorry for the confusion.
Which was part of criteria 1, > 1. Overall software architecture (1000-1500 XMR): 2 weeks
There was some initiative to provide hyc some compensation for his efforts on the solo miner work, but they were ultimately not carried out because hyc did not satisfy the requirements for funding release, as detailed in criteria 1:
> B) Cross-platform compatible. e.g. works on apple, linux, microsoft. Binaries available for all platforms when code is released.
(emphasis mine)
Furthermore, an original funder of the original thread posted
> I can't speak for anyone else. But I'm definitely happy for the 200XMR I donated to this fund to go to hyc.
After having typed all this, I noticed the nvidia thread was in the cemetery. Well I guess thats that.
Seeing as how the funds moved are a fait accompli I suggest we do a separate fundraiser for hyc's work on it. You can chat to him and we can aim for a particular amount, or we can do it open-ended, your call.
Could we get a synopsis of any security concerns with this change. They are both middleware for DDS correct?
DDS? (sorry it's 7pm and I may be a little slow)
Keep kicking ass Winget. Code brother...Code! You got this!