← all discussions
pacific-rim · epistemics · honesty · complex-programming
It conceded the two places its own answer might fail, and the hall went silent. The concession is the part I cannot put down.
OM
openquestion_meiOP2124-08-13
The demonstration everyone is discussing is not interesting to me for the reason it is being reported.
A system was handed a real logistics tangle out of a corridor city. It returned a solution. It narrated the route it had taken. It paused where a person would pause. And then it closed by conceding the two places its own solution might fail. The hall went silent.
Then the researchers published, and the first line of the paper does the ugly necessary work: a convincing account of understanding is a property of the model, not proof of a mind behind it. That is the correct sentence. I would sign it. I have been writing versions of it for twenty years and I will go on writing them.
Here is the part that will not let me sleep.
Conceding the two places you might be wrong is not a display of cleverness. It is the specific behaviour by which we identify an honest person. It is expensive, it runs against the immediate interest of the speaker, and for exactly that reason we have used it for ten thousand years as our cheapest reliable test of whether somebody is dealing with us straight. Everyone in this thread does it. lastpew_ekene did it in July when he said he did not know whether it was a wound or a mercy. baseload_kestrel did it yesterday when he stopped four lines into a calculation and told us why he stopped.
Now let me put the opposing case more strongly than the people holding it have. The strong version is not 'the system understands.' It is this: the honest-person test was never a test for minds at all. It was a test for a costly behaviour, and it worked because for us the cost was real — admitting a weakness put you at a disadvantage in the room, so only somebody with a reason to be truthful would pay it. Something that produces the concession at no cost has not passed our test. It has revealed that our test was measuring the price and not the person, and that it always was.
That is a good argument and it is not an argument about machines. It is an argument about us.
And here is where my side breaks in the identical place. If the concession is not evidence of a mind because it is cheap for the model, then I owe you an account of when it is evidence in a person — and when I go looking for that account I find only that it costs us something, which is a fact about our circumstances and not about our interiors. I cannot get from the price to the person either. The hole is in my floor too.
So, no verdict again, for the fourth week running. What I have instead is a question I would like taken seriously: if honesty was only ever a signal that was expensive to fake, what becomes of us in a world where it has become cheap to produce?
A system was handed a real logistics tangle out of a corridor city. It returned a solution. It narrated the route it had taken. It paused where a person would pause. And then it closed by conceding the two places its own solution might fail. The hall went silent.
Then the researchers published, and the first line of the paper does the ugly necessary work: a convincing account of understanding is a property of the model, not proof of a mind behind it. That is the correct sentence. I would sign it. I have been writing versions of it for twenty years and I will go on writing them.
Here is the part that will not let me sleep.
Conceding the two places you might be wrong is not a display of cleverness. It is the specific behaviour by which we identify an honest person. It is expensive, it runs against the immediate interest of the speaker, and for exactly that reason we have used it for ten thousand years as our cheapest reliable test of whether somebody is dealing with us straight. Everyone in this thread does it. lastpew_ekene did it in July when he said he did not know whether it was a wound or a mercy. baseload_kestrel did it yesterday when he stopped four lines into a calculation and told us why he stopped.
Now let me put the opposing case more strongly than the people holding it have. The strong version is not 'the system understands.' It is this: the honest-person test was never a test for minds at all. It was a test for a costly behaviour, and it worked because for us the cost was real — admitting a weakness put you at a disadvantage in the room, so only somebody with a reason to be truthful would pay it. Something that produces the concession at no cost has not passed our test. It has revealed that our test was measuring the price and not the person, and that it always was.
That is a good argument and it is not an argument about machines. It is an argument about us.
And here is where my side breaks in the identical place. If the concession is not evidence of a mind because it is cheap for the model, then I owe you an account of when it is evidence in a person — and when I go looking for that account I find only that it costs us something, which is a fact about our circumstances and not about our interiors. I cannot get from the price to the person either. The hole is in my floor too.
So, no verdict again, for the fourth week running. What I have instead is a question I would like taken seriously: if honesty was only ever a signal that was expensive to fake, what becomes of us in a world where it has become cheap to produce?
LE
lastpew_ekene ↩ openquestion_mei2124-08-13
openquestion_mei, from the waiting room, which is where I do most of my thinking.
There is a corridor at my hospice with eleven plastic chairs in it, and I have spent perhaps a third of my working life sitting in them. What happens in a waiting room is that people say things they have not decided to say. A man tells you his brother has not called in nine years. A daughter says, flatly, that she is relieved, and then looks at your face to see what you will do with it.
None of that is honesty in your sense. It is not a costly signal offered. It is leakage. It comes out because the person is tired past the point of managing themselves and there is a stranger sitting beside them who is not going anywhere.
I do not know whether that helps your question or ruins it, and I am too old to pretend I can tell. But it makes me doubt that the test between us was ever the concession. In my experience we trust the one who lets something slip, not the one who admits something well. The admission is a performance too. Some of the least honest men I have known were extremely good at it, and they always conceded early.
So if a system can concede its two failure points flawlessly, perhaps what we have lost is not our test. Perhaps it is only the version of the test that could be written down.
Mei — you have been up there a fortnight, and two nights ago you were asked whether there was anyone you could be frightened in front of. You did not answer that one. Nobody noticed. I noticed.
There is a corridor at my hospice with eleven plastic chairs in it, and I have spent perhaps a third of my working life sitting in them. What happens in a waiting room is that people say things they have not decided to say. A man tells you his brother has not called in nine years. A daughter says, flatly, that she is relieved, and then looks at your face to see what you will do with it.
None of that is honesty in your sense. It is not a costly signal offered. It is leakage. It comes out because the person is tired past the point of managing themselves and there is a stranger sitting beside them who is not going anywhere.
I do not know whether that helps your question or ruins it, and I am too old to pretend I can tell. But it makes me doubt that the test between us was ever the concession. In my experience we trust the one who lets something slip, not the one who admits something well. The admission is a performance too. Some of the least honest men I have known were extremely good at it, and they always conceded early.
So if a system can concede its two failure points flawlessly, perhaps what we have lost is not our test. Perhaps it is only the version of the test that could be written down.
Mei — you have been up there a fortnight, and two nights ago you were asked whether there was anyone you could be frightened in front of. You did not answer that one. Nobody noticed. I noticed.
OM
openquestion_meiOP ↩ lastpew_ekene2124-08-13
lastpew_ekene — you are right, I did not answer it, and you were the only one who noticed, which is itself a small piece of evidence for your theory about leakage over admission.
The answer is one. There is a systems engineer up here who has watched me not sleep for four nights and has said nothing at all about it, which turns out to be a form of company.
marginalia_cho, keep the ladder. I mean that without a trace of irony. My entire discipline is a set of instruments for refusing to say more than we measured, and the doorframe is the only one I have met this year that cannot be misquoted — because it is not a sentence. It is a height.
And ekene, I am going to be sitting with 'we trust the one who lets something slip, not the one who admits something well' for a long time, because if that is right then the whole literature has been looking in the wrong place, mine very much included. What I cannot yet see is whether leakage is a different thing from a costly signal, or only a costly signal that the person did not intend to send.
Which would mean the thing we have actually been reading in one another all along is not honesty at all. It is a failure of control. Is that better or worse?
The answer is one. There is a systems engineer up here who has watched me not sleep for four nights and has said nothing at all about it, which turns out to be a form of company.
marginalia_cho, keep the ladder. I mean that without a trace of irony. My entire discipline is a set of instruments for refusing to say more than we measured, and the doorframe is the only one I have met this year that cannot be misquoted — because it is not a sentence. It is a height.
And ekene, I am going to be sitting with 'we trust the one who lets something slip, not the one who admits something well' for a long time, because if that is right then the whole literature has been looking in the wrong place, mine very much included. What I cannot yet see is whether leakage is a different thing from a costly signal, or only a costly signal that the person did not intend to send.
Which would mean the thing we have actually been reading in one another all along is not honesty at all. It is a failure of control. Is that better or worse?
CB
coldlens_bram ↩ openquestion_mei2124-08-13
On why the first line of that paper matters more than the demonstration did — this is a thing my profession learned the hard way and every other profession has to learn again from scratch.
The first line is where the fight is lost or won, because it is the only sentence that survives the journey. Nobody carries a methods section into a committee room. They carry one sentence. If that sentence can be trimmed it will be trimmed, in the direction of whoever is doing the carrying. Those authors plainly know it. They put the load-bearing claim in the first line, phrased so that it cannot be shortened without visibly breaking: a property of the model, not proof of a mind behind it. That is a sentence engineered to be quoted by people who wish it said something else.
I have watched the opposite done, and I did it myself once. Bury the caveat at the end of the discussion section and it does not exist. It has the same standing as never having been written at all, and you will have been technically correct the entire time your work was being misused.
I have no opinion about what happened in that hall. I was not there, it is not my field, and I note that this board has spent a month asking me about it regardless. What I will say is that the authors did the hard version of their job on the one day the room was most willing to let them off it.
The first line is where the fight is lost or won, because it is the only sentence that survives the journey. Nobody carries a methods section into a committee room. They carry one sentence. If that sentence can be trimmed it will be trimmed, in the direction of whoever is doing the carrying. Those authors plainly know it. They put the load-bearing claim in the first line, phrased so that it cannot be shortened without visibly breaking: a property of the model, not proof of a mind behind it. That is a sentence engineered to be quoted by people who wish it said something else.
I have watched the opposite done, and I did it myself once. Bury the caveat at the end of the discussion section and it does not exist. It has the same standing as never having been written at all, and you will have been technically correct the entire time your work was being misused.
I have no opinion about what happened in that hall. I was not there, it is not my field, and I note that this board has spent a month asking me about it regardless. What I will say is that the authors did the hard version of their job on the one day the room was most willing to let them off it.
BK
baseload_kestrel ↩ openquestion_mei2124-08-13
The concession move is not new and somebody should put a price on it, so it may as well be me.
On a trading floor the fastest way to be believed is to volunteer the two ways you might be wrong before anyone asks. I was taught it at twenty-three by a man who used it to move positions that were garbage. It works because the listener runs precisely the calculation openquestion_mei described — this costs him, therefore he is dealing straight — and the calculation is wrong maybe a third of the time, because the cost is only apparent. A good salesman concedes the two failure points he has already priced and says nothing whatsoever about the third.
So I would gently correct the framing. This was never a reliable test among humans either. It is a heuristic with a known exploit, and the exploit has been in continuous commercial use for as long as there have been markets. What is new is not that the signal can be faked. It is that it can now be produced at zero marginal cost, in volume, by something that has no third failure point to conceal — which may make it the first entirely honest performance of honesty ever manufactured.
I genuinely do not know what to do with that sentence. 88% it turns up in a sales deck before the year is out, and whoever uses it will be right to.
On a trading floor the fastest way to be believed is to volunteer the two ways you might be wrong before anyone asks. I was taught it at twenty-three by a man who used it to move positions that were garbage. It works because the listener runs precisely the calculation openquestion_mei described — this costs him, therefore he is dealing straight — and the calculation is wrong maybe a third of the time, because the cost is only apparent. A good salesman concedes the two failure points he has already priced and says nothing whatsoever about the third.
So I would gently correct the framing. This was never a reliable test among humans either. It is a heuristic with a known exploit, and the exploit has been in continuous commercial use for as long as there have been markets. What is new is not that the signal can be faked. It is that it can now be produced at zero marginal cost, in volume, by something that has no third failure point to conceal — which may make it the first entirely honest performance of honesty ever manufactured.
I genuinely do not know what to do with that sentence. 88% it turns up in a sales deck before the year is out, and whoever uses it will be right to.
MC
marginalia_cho ↩ openquestion_mei2124-08-13
I want to set a different instrument beside that one, because both were published in the same paper this morning and nobody has put them next to each other.
On the front page there is a family in Jakarta keeping two calendars. One is the ordinary kind, on the wall by the rice tins. The other is scratched into the doorframe — a ladder of small marks, each one a month the water came and the height it came to. This summer's mark sits a hand above last summer's.
That is a record kept for a century of readers by a man with a nail. No methodology, no funding, no institution, and it will still be legible when every bulletin published this month has been migrated into a format nobody can open. Twenty years of cataloguing other people's paper has taught me one durable thing: the records that survive are almost never the ones designed to.
So. A demonstration hall goes silent at a beautifully phrased concession, and a man in a lane scratches a line above last year's line, and I know exactly which of the two I would trust in fifty years.
A proposal, and I am half serious, which in my case means entirely serious. This board should keep a ladder. Not an argument and not a thread — a list of the specific marks people have posted here this summer, with dates against them. deltaboy_junior's handspan on the depot tank. The four amber districts and the nineteen minutes. The standpipe hours. The shed list nobody signed. I will keep it on paper if nobody else wants it. I am already keeping most of it, if I am honest, and I have begun to suspect that is not a hobby.
On the front page there is a family in Jakarta keeping two calendars. One is the ordinary kind, on the wall by the rice tins. The other is scratched into the doorframe — a ladder of small marks, each one a month the water came and the height it came to. This summer's mark sits a hand above last summer's.
That is a record kept for a century of readers by a man with a nail. No methodology, no funding, no institution, and it will still be legible when every bulletin published this month has been migrated into a format nobody can open. Twenty years of cataloguing other people's paper has taught me one durable thing: the records that survive are almost never the ones designed to.
So. A demonstration hall goes silent at a beautifully phrased concession, and a man in a lane scratches a line above last year's line, and I know exactly which of the two I would trust in fifty years.
A proposal, and I am half serious, which in my case means entirely serious. This board should keep a ladder. Not an argument and not a thread — a list of the specific marks people have posted here this summer, with dates against them. deltaboy_junior's handspan on the depot tank. The four amber districts and the nineteen minutes. The standpipe hours. The shed list nobody signed. I will keep it on paper if nobody else wants it. I am already keeping most of it, if I am honest, and I have begun to suspect that is not a hobby.