Dialog*--characters talking--is one of the key parts of almost any story you will write. There are specific conventions around writing dialog, and if you demonstrate that you don't know them, it will be difficult for readers who do know them to take you seriously (or concentrate on the story).
*I'll use the American spelling for "dialog," because it's shorter.
Let's start out by defining a term. I'll be talking a lot about "dialog tags" or "tagging" dialog. You can think of this as like a luggage tag: it tells you who the dialog belongs to. The basic dialog tag is "he said" or "she said" or "said Ashley" (if your character is called Ashley).
If you don't put in enough dialog tags, your readers will lose their way (especially if your characters sound similar to one another and there's nothing in what they say that indicates who is speaking). If your reader has to go back and count on their fingers from the last dialog tag to work out who said what, you've lost their immersion in the story.
If you put in too many dialog tags, on the other hand, it slows down the exchange, and your readers may feel that you're treating them like idiots. If there are only two people in the scene, you don't need to tag very often to keep it clear who said what. We know they're alternating.
Your beta readers should tell you if you're tagging too often or too seldom.
"Beats" are an alternative to dialog tags. A beat is an action that the character takes during dialog:
"So tell me about your date." John took a long slurp on his milkshake.
"I turned up late, and that was the best thing about the evening."
The convention is that if the beat occurs in the same paragraph as the line of dialog, then both the dialog and the action belong to the same character. As the example above stands, we assume that John is speaking the first line and someone else is speaking the second line. If we moved the beat from after the first line of dialog to before the second, we would assume that John was the second speaker.
Beats have a few functions. They prevent the scene from feeling like disembodied heads talking in a white room; they can provide characterisation and setting detail; they can convey emotion; and they provide a touch of variety, a relief from "he said," "she said".
While I'm talking about varying tags, there are different schools of thought on how much to vary them. Many writing teachers are firm that you should only ever use "said," because this enables the dialog tags to disappear from the reader's awareness. They become almost subliminal cues that don't draw attention to themselves. Even "asked" and "replied" are a step too far for some experts. The reader can tell from the punctuation that a question was asked and replied to, they say. You don't need to make it explicit.
A much smaller group of teachers recommend what are sometimes called "said bookisms," fancier synonyms for "said" like "ranted," "uttered," "declared," or "asseverated". These create a specific style, old-fashioned, formal, and ornate. If that's the style you're going for, and you have the writing chops to pull it off (few people do), then by all means use them. If you're attempting "invisible" style, though--and my advice is that you learn to write invisibly first--don't go near them.
I hold a middle view. I think it's fine to use words like "whispered" or "muttered" in place of "said" from time to time. I'll occasionally throw in an "asked" or "replied". But none of my characters "elucidate" unless I'm imitating a particular style.
There are two parts to a tag: the "said" (or, possibly, equivalent), and the identifier, the name or pronoun or other reference to who is speaking. Within limits, you can vary which comes first.
"Why did it have to be snakes?" he said.
"Why did it have to be snakes?" said Indiana Jones.
"Why did it have to be snakes?" Indiana Jones said.
"Why did it have to be snakes?" said the archaeologist.
"Why did it have to be snakes?" the archaeologist said.
You'll notice one construction missing from that list:
"Why did it have to be snakes?" said he.
"Said he" is old-fashioned and draws attention to itself, placing the emphasis on the "he". Use it sparingly, and only for deliberate effect, if you use it at all. Oddly enough, in the other examples, placing the identifier first puts a subtle emphasis on the identifier (in my opinion; you may have a different reaction). Of all of these examples, the first two are the most neutral and invisible, which is usually what you want.
If you're using a pronoun, make sure it's clear what the reference is. This is a general rule, not just for dialog. "He" refers to the last male character specified, "she" to the last female character specified, and especially if you've changed your sentence round during editing, the reference can become ambiguous. (Watch out for "it" and "this" as well. They are even more likely to become ambiguous.)
I read one book where the author had several times used "he," in a scene with two male characters, to refer to the character who was not the most recent person named. It was disorienting; each time, I thought that it was the other character at first (the most recently named character, as would be usual), then figured out that it wasn't, glanced back to see if I'd missed something, and finally concluded that it was a mistake. By this time, any immersion in the story was well and truly shattered.
Tagging Before, During and After
You can tag in three different places: before the dialog, during the dialog, or after the dialog.
Tagging before lets the reader know who's about to speak, which can be important to how they read it, but it also draws extra attention to the identity of the speaker, and that may or may not be where you want to put the emphasis.
The colonel said, "Load of nonsense. No such thing."
Tagging during the dialog allows you to break at a significant moment in order to emphasise something or make it more dramatic.
"Snakes," said Indiana Jones. "Why did it have to be snakes?"
Make sure your break comes at a natural place, between sentences or at some other grammatical boundary, or it will sound awkward and risk confusing the reader. Bad example:
"You crack me so consistently," said Fat Tony, "up".
There, you'd be better served by using an ellipsis to indicate the pause, especially as the grammar is odd to begin with.
Tagging after the dialog is most neutral and invisible:
"Get that aardvark out of here!" said Aunt Nora.
I've been showing you examples of punctuating dialog without drawing attention to it. Now it's time to think about the conventions. I've seen authors get every one of these wrong at one time or another.
A line of dialog and its tag form a single sentence. This sentence follows all the rules of normal sentences--begins with a capital letter, ends with either a question mark, exclamation mark, or period--but with a few twists.
Most obviously, there are quotation marks around the part that the character said. Be careful with these. Not closing them is a very common error, not opening them a less common one (but I have seen it). That usually happens because you're typing too fast.
The American convention is to use double quotation marks, with any quote-within-a-quote getting single quotation marks, while the British convention is the other way around. Even though I live in New Zealand, which is a British Commonwealth country, I use the American convention. (I blame excessive exposure to American books at a formative age.)
The points of transition, when you come in and out of the quotation, are important, and marked by particular punctuation. Let's look at that.
Doug said, "Ease it gently into the cradle. Gently!"
A tag before the dialog typically ends with a comma. You can get away with leaving the comma out, but the effect is to increase the pace of the sentence. I usually put the comma in, so that I can leave it out if I want the pace to seem brisker.
The dialog then starts with a quotation mark and a capital letter (to mark the start of a sentence). At the end of the dialog, after the sentence-closing punctuation, we get a closing quotation mark. If there are sentence boundaries inside the dialog, they are marked as usual.
"Gently," said Doug, "we don't want to crack it."
A tag in the middle of the dialog is part of a whole sentence. You need a comma before the first closing quotation mark, and another after the tag. Those aren't optional. Because the start of the second part here (the word "we") comes in the middle of a sentence--it's not starting either the whole sentence or a sentence in the dialog--it doesn't get capitalised. The closing punctuation (here, a period) goes inside the quotation marks.
"We don't want to crack it," he said.
A tag at the end of the dialog needs a comma before the closing quotation mark, in place of the period that would usually finish this sentence (but see below if it's a question or exclamation).
The sentence isn't over until the tag is finished. Do not capitalise "he" in a sentence like this, as if the tag was its own sentence. It isn't, even though the dialog is a complete sentence grammatically. The dialog and the tag are one sentence for purposes of punctuation. I sometimes see this error from inexperienced writers:
"We don't want to crack it." He said.
However, if you use a beat, it's a complete sentence by itself:
"We don't want to crack it." He frowned at me.
You can also put a tag and a beat together, in which case the tag rules apply:
"We don't want to crack it," he said, frowning at me.
What if the end of the dialog has a question mark or exclamation mark, though? In the examples above where a tag follows the sentence, we've turned the period that would usually be there into a comma. We don't do that with a question mark or exclamation mark--they remain themselves. There's no comma, but otherwise the punctuation of the sentence is the same:
"Do you want to crack it?" he said.
"Don't crack it!" he said.
Note, in particular, that "he" is not capitalised. Again, the tag is not a sentence by itself.
Speech Continuing Across Paragraphs
There's another dialog punctuation convention that is very well known, but I've seen at least two authors get it wrong in different ways, so I'll mention it. If the same speaker continues in the next paragraph (without any tag or beat between), you leave off the closing quotation mark in the previous paragraph, like this:
"I never knew for sure how it happened, but I have a few guesses.
"Firstly, the chameleon was somewhere in that room."
The first way I've seen this done wrong is putting in the closing quotation mark after the first paragraph, which, by convention, indicates a change of speaker. In the story I was reading, the speaker hadn't changed. It was especially confusing because that author didn't make much use of tags or beats, so it was hard to tell when the speaker did switch.
The second way I've seen it done wrong is not closing the quotation if you insert a beat or tag between the paragraphs. You need the closing quotation mark to separate the dialog from the beat or tag in this case.
Let's Eat Grandma
One more thing about dialog before we get into commas. It's a comma convention, so it could go in either chapter, but there are plenty of other comma rules, and I want to discuss this one here.
I call this the "Let's eat Grandma" rule, because of the most famous example. These two sentences are different:
"Let's eat Grandma."
"Let's eat, Grandma."
The first is talking about Grandma and proposing cannibalism. The second is talking to Grandma and proposing dinner.
To me, there are few more glaring marks of an amateur writer than ignorance of this rule (which is called the vocative comma rule). Always, without exception, use a comma before a term of address in dialog:
"What up, homie?"
"Make sure you take your aardvark with you, my friend."
"I want to get to know you better, John."
This rule only applies to terms of address, though:
"I'm fascinated by the captain."
"I want to get to know John better."
In those examples, you're talking about the person, not to them, so there's no comma.
The rule has a second half: always use a comma after a term of address, as well:
"John, come and look at this aardvark."
"Captain, I canna give ye any more power!"
And if the term of address comes in the middle of a sentence, it gets commas both before and after:
"Dammit, Jim, I'm a doctor, not a grammarian!"
Leaving out these commas around terms of address makes your dialog seem breathless and rushed. You can use it deliberately to create that effect, of course (I've done so), but in general, show your readers that you know the rule by putting in the commas.