Wolf0 posted this 9 years ago
This is a proposal that I'm going to do in parts, due to the nature of it - I won't know how much work there is to do, or what milestones to set, until the first step is done. So, first, let me describe the problem XMR has.
All PoW mined cryptocurrencies today face the possibility of 50%+ attacks. While they're called "50%+ attacks," or "51% attacks" - this is a misnomer. Having greater than 50% only ENSURES your attack will succeed. Success can be likely with even 30% of the network hashrate, especially if reversing few confirmations. The attacker has infinite tries, and only needs to get lucky once. Now, a single entity amassing this much hashpower becomes more and more unlikely as the currency grows, but we face more of a threat from the pools the miners use.
XMR currently has what is referred to as "stratum," but is nothing of the sort. It is effectively Getwork over TCP. Now, the scalability issues aren't much of an issue here as they were with Bitcoin - as XMR miners tend to mine in KH/s at the most, miners will hardly run out of nonces to try. But the greater threat here is that the miner knows NOTHING about what they're hashing. They will blindly solve hashes for the pool, even if the block being mined is for attacking the network. Additionally, block creation is not decentralized in any way - the pool chooses the transactions which go into the blocks, as well - this is a disaster for decentralization. On top of all of this, the current mining infrastructure code is so badly put together that it will break in the future - exactly when, I've not calculated, but XMR's block header size, and the location of the nonce, is not constant. When the timestamp grows by a byte, and it will, most miners will cease to mine valid blocks.
Clearly something needs to be done, but what? I propose a heavy modification of Getblocktemplate (GBT) from Bitcoin. The only thing that can be taken from it is the basic ideas, because of how different XMR and BTC are, but the goals are such that miners who run daemons/wallets will check the pool's work against the network, as well as create their very own custom blocks. The pool will still be accepting shares, the same as before, but instead of checking with a block header they sent you, they will check your block proposal and ensure it pays the pool. This means miners may include their own transactions, their own coinbase signatures, and examine every detail of the pool's work that it's handing out. If even a small portion of miners run daemons/wallets, allowing the miner to police the pool, then they can warn everyone else if a pool becomes malicious. Since attacks take time and sustained hashrate, this guts the effectiveness of abusing innocent miners' hash for a malicious attack.
Now, in order to determine how to proceed, I need to do a LOT of research. I need to know the exact layout of blocks, transactions, and how blocks are built from the ground up. This is because block creation will have to be implemented in the miner - I'll also need to consider pool side verification of shares, and the best way to implement communication between the miner and pools. I'll have to bring that all together to create a technical specification for the new protocol - for everyone to be able to implement. After this is complete, I probably will post another thread proposing the relevant development.
So, is this something the community would like to pursue?
EDIT:
As a footnote, here's a comment I posted with proof of the mining code breaking:
For you to look, here's the relevant details:
Here's where the miner hardcodes that the nonce is 39 bytes into the block header: https://github.com/lucasjones/cpuminer-multi/blob/master/cpu-miner.c#L1071
And here's the block header format, showing that before the nonce, there are three variable-sized integers: https://github.com/monero-project/bitmonero/blob/master/src/cryptonote_core/cryptonote_basic.h#L289
Really, all the current infrastructure has had corners cut at every possible opportunity.
I propose 750 XMR in total over two milestones here. The first milestone will be supporting documentation, and will help XMR in more than one way: more of the data structures and the internal workings of Monero will be documented for people wishing to study it for other purposes than mining. This will include the format of blocks, which requires the format of Monero's variable-sized integers to be documented as well. Also needed for this milestone is documentation on difficulty (difficulty to target and back, calculating average network hashrate from difficulty, this sort of thing), and detail on creating AT LEAST the coinbase transaction (the one that pays the miner.) Full documentation on transactions is not necessary, but would be nice. These entries are to be posted on monero.wikia.com.
The second milestone I propose is the details of the protocol between the pool and the miner. This will include the format of the information and what sorts of information need to be passed between the miner and pool. It will also detail rules for communication between them, and detail how to construct block proposals for the pool, etc.
450 XMR for the first milestone, as it requires the most research and time put in, 300 XMR for the second.
The lack of activity on this project has put these funds in danger of being re-appropriated.
If any funders significantly disagree, please voice your concern. Funds will probably be moved to pay for hycs work on the solo nvidia miner (which, imo, is well deserved and sort of also gives a nod to the fact that he's, in general, doing awesome monero developing and stuff)
I support this.
My suggestion would be for you to put an initial effort into this research, say 100 hours or whatever (I'm thumb-sucking). Coming off the back of that research (ie. the deliverable) is a document / write-up that has an overview of how it currently works, and then details how it will work in future. This should be as detailed as possible, as future pool implementations will use it as a technical reference, and it will also be invaluable if you get hit by a bus. If you want part-payments throughout the process then I'd suggest you break the write-up into sections, and have a milestone per section.
Also we can have the members of the Monero Research Lab provide input, and it can go out as an MRL research bulletin with you as the author, so that's nice.
This is EXACTLY what I'd planned. The first set of milestones would be in documentation, leading up to the full specification. If the members of the MRL are familiar with the code, they might be able to help me with the format of some of the data structures.
No one is really familiar with the code enough to just rattle off data structure formats, other than maybe the people who worked on pool/miner stuff last year, as we can see they didn't do a great job anyway. To get the nitty gritty will require using the code as a reference. Some of tewinget's documentation may be helpful, assuming he has covered the relevant portions of the code by now.
Responding more generally to the OP:
Someone quoted moneromooo on the bitcointalk thread as saying that the timestamp issue doesn't arise for 1000 years (I have not confirmed) so while that was worth raising as an abstract concern, it seems to be a non-issue in practice.
Overall I'm generally supporting of improving the pool infrastructure but I would urge sponsors to consider carefully how highly this ranks among overall development priorities for the project and avoid overcommitting resources to something that is quite open ended and that even when completed may find adoption of such an upgrade by pools and miners to be slow or nonexistent.
Which is to say, we have pools. They're not perfect, but they do work. If you look at the development goals document, or even consider things we could do as a project that aren't even on there, that includes many things we don't have at all. I would hate to get to a situation where money was spent on this (potentially a very significant amount to get past the first phases and to completion) and when the time comes to spend money on valuable functions that don't currently exist, and qualified developers are available to do that work if funding is available, sponsors feel tapped out. It is easy to suggest with a sense of idealistic optimism that money will always be there for good work, but in reality resources are likely to be limited, especially if the market continues to not cooperate in valuing this coin more.
Finally, some within the community don't even support pooling at all and view it as inherently harmful (and suggest that a development priority should be pool resistance measures). Investing in the pooling infrastructure means either dropping that approach or investing in two contradictory directions. The latter is not necessarily bad as an diversified approach to development (it just means more and better options, even if some are later unused) but again the question of limited resources applies.
But if people are confident that resources are not being overextended then by all means I support the proposed ideas taken on their own.
My beliefs coincide with Smooth on this one. This overhaul may very well be needed, but its importance may fall behind some other necessary projects that we should not want to deviate funds from.
I would suggest that if this project gets going, that in the research portion, it is looked into as to the impact of the overhaul, and as to whether or not the importance exceeds other projects.
I believe pool mining is necessary, but like others have stated, it isn't perfect. I would rather p2pool or something similar be used more than ordinary pools as well.
I think Wolf0 is a great developer and I am glad he is working with Monero. If he and the rest of the community deem this project to be a necessity, and that it should be worked on sooner than later, then I will happily contribute.
I didn't know when the timestamp issue would occur, but it serves its purpose in my proposal as an example of the bad craftsmanship and corners cut in the current pooling infrastructure.
I knew it would require studying the code - I also plan to document what I learn on the way, for others. However, all I wanted to say was it would be nice to have someone with experience who knows where the stuff is.
Further, the idea of not having pooling at all is, quite frankly, stupid. A successful coin will have an amazingly high difficulty, and small miners will stop mining the coin when they see nothing for weeks to months. This proposal is the best of both worlds - individuals are creating blocks, while the only shared thing is the payout.
On the subject of priority limited funds for good work - I completely agree.
For what its worth, I agree with the sentiment of funder burnout, but I don't know when that will occur, and when that occurs depends on how large the funding efforts are. If this one rolls through, we would currently have 2 large fundraising efforts. 18k XMR to Moneromoo, and a currently unknown amount to Wolf.
I think, in general, part of the situation is a lot of Monerians really want to see things develop, so we're excited to fund development of anything, and we're excited to fund developers to spend time on the Monero code. For reasons unknown, Monero isn't attracting talent left and right, and even previous contributors presumably don't have the time to become community-funded developers (with the exception of moneromooo and tewinget). Thus, to brush aside interest in developing because it doesn't align with the research and development goals doesn't seem right.
And what is particularly relevant for this effort is inertia / momentum. What I mean by that is that if, for whatever reason, this effort is tabled for now and the current system keeps chugging along, its possible it will get to the point where it doesn't change simply because the way things are working is "fine". FFS, look at bitcoin - the pool infrastructure could have been designed such that individual miners are validating their own blockchain, but what got hacked together with probably very little foresight was the current infrastructure, and now its the norm that "pooling" means 1 node defines a block, and participants of the pool submit solutions to that block. Thus, IMO, the definition of a pool as a mining cooperative is wrong. In a true cooperative, individuals would have a say.
Thus, yes, we do have pools, and they are not perfect. But while the community / network / infrastructure is small, we have the chance to make things better before it becomes unwieldy to effect change.
And I concur... I also am in the camp that pool resistance should be a priority, but Wolf repeatedly reminds me that this approach would be more of a true cooperative mining architecture than the current "pooling" architecture. I mean, in essence, if we could have a p2pool type thing, that would be best. I think keeping a centralized pool server might somehow fix the problem encountered in p2pool with latency or whatever, but hopefully you get my drift.
but, as always, take this all with a grain of salt because I just came into this space 1 year ago, so there's that.
> FFS, look at bitcoin - the pool infrastructure could have been designed such that individual miners are validating their own blockchain, but what got hacked together with probably very little foresight was the current infrastructure
That's not what happened and what did happen is relevant. Getblocktemplate was designed around the same time as stratum, maybe even a bit earlier. Miners and pools did not adopt it because they didn't care and just wanted a simpler and lower overhead solution, with no concern whatsoever for validating anything.
The more recent developments with SPV mining shows miners and pools will go beyond even that, and cut any corner to save costs even if it means greatly compromising the security of the chain.
And all that is in the case of Bitcoin, which doesn't have the additional issue of alts where miners frequently just move on to the next coin, giving them even less reason to care about the health of any one coin.
You really can't expect buy in from miners on making the chain secure. They will do the minimum that allows them to get paid, and will search for ways to do even less.
> That's not what happened and what did happen is relevant.
thanks for the clarification. us 3rd wave crypto folks just dont know some things.
Supported.
