Wolf0 posted this 8 years ago
This is a proposal that I'm going to do in parts, due to the nature of it - I won't know how much work there is to do, or what milestones to set, until the first step is done. So, first, let me describe the problem XMR has.
All PoW mined cryptocurrencies today face the possibility of 50%+ attacks. While they're called "50%+ attacks," or "51% attacks" - this is a misnomer. Having greater than 50% only ENSURES your attack will succeed. Success can be likely with even 30% of the network hashrate, especially if reversing few confirmations. The attacker has infinite tries, and only needs to get lucky once. Now, a single entity amassing this much hashpower becomes more and more unlikely as the currency grows, but we face more of a threat from the pools the miners use.
XMR currently has what is referred to as "stratum," but is nothing of the sort. It is effectively Getwork over TCP. Now, the scalability issues aren't much of an issue here as they were with Bitcoin - as XMR miners tend to mine in KH/s at the most, miners will hardly run out of nonces to try. But the greater threat here is that the miner knows NOTHING about what they're hashing. They will blindly solve hashes for the pool, even if the block being mined is for attacking the network. Additionally, block creation is not decentralized in any way - the pool chooses the transactions which go into the blocks, as well - this is a disaster for decentralization. On top of all of this, the current mining infrastructure code is so badly put together that it will break in the future - exactly when, I've not calculated, but XMR's block header size, and the location of the nonce, is not constant. When the timestamp grows by a byte, and it will, most miners will cease to mine valid blocks.
Clearly something needs to be done, but what? I propose a heavy modification of Getblocktemplate (GBT) from Bitcoin. The only thing that can be taken from it is the basic ideas, because of how different XMR and BTC are, but the goals are such that miners who run daemons/wallets will check the pool's work against the network, as well as create their very own custom blocks. The pool will still be accepting shares, the same as before, but instead of checking with a block header they sent you, they will check your block proposal and ensure it pays the pool. This means miners may include their own transactions, their own coinbase signatures, and examine every detail of the pool's work that it's handing out. If even a small portion of miners run daemons/wallets, allowing the miner to police the pool, then they can warn everyone else if a pool becomes malicious. Since attacks take time and sustained hashrate, this guts the effectiveness of abusing innocent miners' hash for a malicious attack.
Now, in order to determine how to proceed, I need to do a LOT of research. I need to know the exact layout of blocks, transactions, and how blocks are built from the ground up. This is because block creation will have to be implemented in the miner - I'll also need to consider pool side verification of shares, and the best way to implement communication between the miner and pools. I'll have to bring that all together to create a technical specification for the new protocol - for everyone to be able to implement. After this is complete, I probably will post another thread proposing the relevant development.
So, is this something the community would like to pursue?
EDIT:
As a footnote, here's a comment I posted with proof of the mining code breaking:
For you to look, here's the relevant details:
Here's where the miner hardcodes that the nonce is 39 bytes into the block header: https://github.com/lucasjones/cpuminer-multi/blob/master/cpu-miner.c#L1071
And here's the block header format, showing that before the nonce, there are three variable-sized integers: https://github.com/monero-project/bitmonero/blob/master/src/cryptonote_core/cryptonote_basic.h#L289
Really, all the current infrastructure has had corners cut at every possible opportunity.
I propose 750 XMR in total over two milestones here. The first milestone will be supporting documentation, and will help XMR in more than one way: more of the data structures and the internal workings of Monero will be documented for people wishing to study it for other purposes than mining. This will include the format of blocks, which requires the format of Monero's variable-sized integers to be documented as well. Also needed for this milestone is documentation on difficulty (difficulty to target and back, calculating average network hashrate from difficulty, this sort of thing), and detail on creating AT LEAST the coinbase transaction (the one that pays the miner.) Full documentation on transactions is not necessary, but would be nice. These entries are to be posted on monero.wikia.com.
The second milestone I propose is the details of the protocol between the pool and the miner. This will include the format of the information and what sorts of information need to be passed between the miner and pool. It will also detail rules for communication between them, and detail how to construct block proposals for the pool, etc.
450 XMR for the first milestone, as it requires the most research and time put in, 300 XMR for the second.
The lack of activity on this project has put these funds in danger of being re-appropriated.
If any funders significantly disagree, please voice your concern. Funds will probably be moved to pay for hycs work on the solo nvidia miner (which, imo, is well deserved and sort of also gives a nod to the fact that he's, in general, doing awesome monero developing and stuff)
As I've stated elsewhere, I think the mining infrastructure of monero is currently underdeveloped, and the less effort put it into it now, the more likely it is to run away and turn bitcoin-like or to break entirely (as pointed out by wolf above). Thus, I think its something that should be pursued.
For you to look, here's the relevant details:
Here's where the miner hardcodes that the nonce is 39 bytes into the block header: https://github.com/lucasjones/cpuminer-multi/blob/master/cpu-miner.c#L1071
And here's the block header format, showing that before the nonce, there are three variable-sized integers: https://github.com/monero-project/bitmonero/blob/master/src/cryptonote_core/cryptonote_basic.h#L289
Really, all the current infrastructure has had corners cut at every possible opportunity.
I support this.
My suggestion would be for you to put an initial effort into this research, say 100 hours or whatever (I'm thumb-sucking). Coming off the back of that research (ie. the deliverable) is a document / write-up that has an overview of how it currently works, and then details how it will work in future. This should be as detailed as possible, as future pool implementations will use it as a technical reference, and it will also be invaluable if you get hit by a bus. If you want part-payments throughout the process then I'd suggest you break the write-up into sections, and have a milestone per section.
Also we can have the members of the Monero Research Lab provide input, and it can go out as an MRL research bulletin with you as the author, so that's nice.
This is EXACTLY what I'd planned. The first set of milestones would be in documentation, leading up to the full specification. If the members of the MRL are familiar with the code, they might be able to help me with the format of some of the data structures.
Supported.
No one is really familiar with the code enough to just rattle off data structure formats, other than maybe the people who worked on pool/miner stuff last year, as we can see they didn't do a great job anyway. To get the nitty gritty will require using the code as a reference. Some of tewinget's documentation may be helpful, assuming he has covered the relevant portions of the code by now.
Responding more generally to the OP:
Someone quoted moneromooo on the bitcointalk thread as saying that the timestamp issue doesn't arise for 1000 years (I have not confirmed) so while that was worth raising as an abstract concern, it seems to be a non-issue in practice.
Overall I'm generally supporting of improving the pool infrastructure but I would urge sponsors to consider carefully how highly this ranks among overall development priorities for the project and avoid overcommitting resources to something that is quite open ended and that even when completed may find adoption of such an upgrade by pools and miners to be slow or nonexistent.
Which is to say, we have pools. They're not perfect, but they do work. If you look at the development goals document, or even consider things we could do as a project that aren't even on there, that includes many things we don't have at all. I would hate to get to a situation where money was spent on this (potentially a very significant amount to get past the first phases and to completion) and when the time comes to spend money on valuable functions that don't currently exist, and qualified developers are available to do that work if funding is available, sponsors feel tapped out. It is easy to suggest with a sense of idealistic optimism that money will always be there for good work, but in reality resources are likely to be limited, especially if the market continues to not cooperate in valuing this coin more.
Finally, some within the community don't even support pooling at all and view it as inherently harmful (and suggest that a development priority should be pool resistance measures). Investing in the pooling infrastructure means either dropping that approach or investing in two contradictory directions. The latter is not necessarily bad as an diversified approach to development (it just means more and better options, even if some are later unused) but again the question of limited resources applies.
But if people are confident that resources are not being overextended then by all means I support the proposed ideas taken on their own.
I didn't know when the timestamp issue would occur, but it serves its purpose in my proposal as an example of the bad craftsmanship and corners cut in the current pooling infrastructure.
I knew it would require studying the code - I also plan to document what I learn on the way, for others. However, all I wanted to say was it would be nice to have someone with experience who knows where the stuff is.
Further, the idea of not having pooling at all is, quite frankly, stupid. A successful coin will have an amazingly high difficulty, and small miners will stop mining the coin when they see nothing for weeks to months. This proposal is the best of both worlds - individuals are creating blocks, while the only shared thing is the payout.
On the subject of priority limited funds for good work - I completely agree.
