Flowdock to Discord or: How I Learned to Stop Worrying and Love Bishop
In The Beginning, There Was Flowdock
The original team behind Keep has been using Flowdock since well before Keep’s launch in April 2020. At the time we were originally making these decisions, we knew we needed a platform to help us meet the following needs:
- Organize information into categories
- Separate out side conversations
- Let people link to old conversations
- Structure reply hierarchy
- Search old messages
- Fine-tune notifications
- Have synchronous tools (“so-and-so is typing”, “so-and-so has seen your message”, etc)
- Have message formatting (markdown, emoji, gifs, embedding, etc)
- Own the underlying message data
- Product Support
At the time, The Chat-Tool-Wars were just getting started. There was Campfire, HipChat, Slack, Flowdock, IRC, plain old email, Skype, Zulip, and others that I'm sure I'm forgetting. Unfortunately, none of the programs checked every box!
Most modern chat programs (and forums) have the concept of "chat channels"; which is essentially the ability to categorize by subject. Chat channels let you keep your discussions about business development in one channel and your debates over solidity gas optimizations in another. If everyone is chatting in one single, shared channel about everything, things quickly devolve into total chaos. So, having a single channel for discussion was never an option.
And so, we set up what we felt would be a few, minimal, primary channels that would broadly cover the most common topics. Unfortunately, this did not prove to be an adequate solution. What happens when you have a chat channel for "tech", and you're using it to have a conversation about pull request review standards while two other people on your team are having a different conversation about some client code retry logic? Still chaos; though granted, slightly less chaos. The messages for both topics start to get interwoven and are incredibly difficult to parse through.
However, if you start creating an individual channel for every potential topic of conversation, then you eventually arrive at total chaos as well. We began creating new channels every time we needed to talk in-depth about a topic. Whenever we found ourselves knee-deep in a topic we didn't expect to be in, someone would say "let's spin this out into its own channel". At this point, we find ourselves in channel-hell, where our slack (or whatever tool we were using at the time) server looked like a huge list of abandoned and short-lived channels. Some of them might be designated as the "main" channels, and when some of us got sufficiently frustrated at the clutter we wrote a slack bot to archive dead channels for us.
Flowdock, and later Slack and Discord, try to solve this problem (with varying degrees of success) with what’s known as the thread tool. In Flowdock's implementation, you either type into the main channel window - which creates ‘unthreaded’ messages - or you click on a message before you type your response. This splits your view into two panes: all messages for the channel on the left, and all messages associated with the original message (the thread) you just clicked on to the right; each with their associated chat box. Typing in the right chat box will add your message to both panes, since the right is a subset of all messages within the channel.
The general workflow for everyone posting messages then became as follows: one would write their initial message in a channel. Then, if one had subsequent messages related to that topic, one would click on the initial message, and start a thread.
The general workflow for everyone responding became as follows: one would read through a channel for new, interesting messages. If you saw one, you clicked on it to review the thread for just those messages; this helped ‘remove the noise’ of other topics being discussed within the same channel. Then, if one wanted to add to the conversation, one responded in the thread. One should always respond within a thread since a thread response is added to both the thread context, as well as resurfaced to the channel.
Given that the rest of the feature set was roughly comparable, this differentiator was enough to convince us to choose Flowdock.
So Long, Old Friend
As the years went by, Flowdock seemed to go on life support. New features and updates slowed and then stopped. The marketing team stopped marketing. The product changed hands multiple times. The competition caught up. Discord (which I've been using since beta in ~2015; lifelong WoW player) started making strides, and eventually pivoted away from targeting just gamers to targeting communities more broadly. Eventually, both Discord and Slack implemented threads.
At present, the Keep community, the NuCypher community, both DAOs, the Threshold community, its DAO, and basically the DeFi community in general runs on Discord. People grew tired of Flowdock's lackluster search, its clunky notifications, and most new people who were becoming involved with our community weren't familiar with Flowdock but were already familiar with Discord.
We switched to an internal Discord server with the goal of using a communication tool that puts us closer to our community and users, and hopefully encourages us to engage with that community more often. We loved that discord had built-in voice+video channels. We loved that you can control notifications on the level of individual channels. We loved the search functionality.
The decision proved timely! On April 3rd, 2022 Flowdock announced that it would be shutting down on August 15th, 2022. So long, old friend; you will be missed.
Pain Points
The first pain point with Discord is that, unlike in Flowdock, creating threads is high-friction. If you want to create a thread in Flowdock, you just type a message in the left-side chat box. If you want to create a thread in Discord you need to press the ‘new thread’ button, then you are required to give it a name (naming a thread is harder than you might think), and then enter your message. If you want to add to a thread, you have to @-mention the relevant people before you are able to add
messages.
The second pain point is when you create a new thread in Discord no one receives a notification of the message unless you @them in the thread specifically.
The third pain point with threads on Discord is if a thread hasn't seen recent activity, it automatically archives (disappears from the thread list; nobody can post new messages, though it is still searchable). Folks used to the Flowdock-style mode of communication will just see an unhelpful list of thread titles in a channel, start a new thread, talk in it, and then wait for responses.
If someone actually does respond in a thread, no one else on the team knows that this response happens because unless you've specifically @-mentioned, the response is hidden from you (you're either "in" the thread or "out" of the thread on a client level).
So, the result of switching to discord has been three-fold: First, thread usage plummeted. People started using Discord like you would use Slack or IRC in the days of yore, intermingling all of their messages chaotically. Second, async Discord text communication plummeted. Since it was so hard to keep everything straight in text, folks resorted to talking through complex topics in calls. Third, information became partitioned. People were still doing lots of writing in threads, but almost everyone had almost no awareness of almost any of it.
So, what is to be done?
Solution 1: Just Write Code?
First, we needed some sort of process around new thread creation. There's a gradient between Flowdock's "all messages belong in a thread" and IRC's "what's a thread?", and we settled on the following heuristic: "If you ask a question and get an answer, that's fine to have as chatter in the main channel. If the conversation goes further than that, or you think it will, make a thread and copy in the context."
Second, we needed some way to rustle up all everyone to every thread, every time. Why not make the first message in every thread @everyone? Nope. Doesn't work. I don't know why. But, if we assign everyone a role like Keep, and then write @Keep at the top of every thread, then now everyone gets added to every thread!
Combined, this solves the "almost everyone had no awareness of any of the information in threads" problem; but creates two new ones.
- Information and Notification overload
- It's a brittle human process that's easy to forget to do
In order to solve the first problem, we can play with the notification settings a little and use a bit of a Discord hack. To cut down on noise (and also for general mental health), changing any moderately busy server to "only @mentions" is a game-changer. You'll still get an unread badge for any servers, channels, and threads with unread messages, but won't get an alert when new messages are written unless you're @mentioned.
We also take advantage of a quirk (bug? feature?) in Discord where a message that is edited into an @mention doesn't trigger the notification, but does have all of the other properties. So, to quietly and politely add everyone to a thread, you write edit, and then edit that to @Keep; fingers-crossed that this doesn't go away.
As to the second problem. Well, we're engineers, so when adopting technology gives us problems, we solve those problems by creating more or adopting more technology. (We'll hit the bottom eventually, right?.. right?)
Enter: Bishop
Bishop is our newest team member and is our friendly discord bot written in discord.js. Since inception he's taken up some additional duties, but relevant to this post, he automates three things around the threading process:
- He nudges people engaging in a conversation in a channel to make a thread
- He automates the aforementioned "edited
@Keepin every thread" maneuver and sets it to auto-archive after a week - He asks the creator of a thread that hasn't seen activity in 4 business days if they want to archive it
The Nudge
client.on('messageCreate', async message => {
if (!!message.reference && !!message.reference.messageId) {
const channel = await client.channels.fetch(message.reference.channelId)
if (!channel.isThread()) {
const referenceMessage = await channel.messages.fetch(message.reference.messageId)
if (!!referenceMessage.reference) {
message.react(EMOJI)
}
}
}
});
Here’s that explanation again in non-Dev English: whenever we see a message that isn't already in a thread, we want to see if that message is a reply to a reply. If it is, we want to react with the pre-specified emoji (we just uploaded a screenshot of the new thread icon).
So now Bishop watches for replies-to-replies, and as soon as he sees one he says "hey wait a minute, the heuristic was 'If you ask a question and get an answer, that's fine to have as chatter in the main channel. If the conversation goes further than that, or you think it will, make a thread and copy in the context.'" and nudges them to do that with a little emoji reaction of a thread.
Seems to be working so far!
The Quiet Maneuver
client.on('threadCreate', async thread => {
if (thread.ownerId !== client.user.id) {
await thread.join()
const placeholder = await thread.send("<placeholder>")
await placeholder.edit("<@&" + ROLE + ">")
if (thread.autoArchiveDuration < sevenDaysInMinutes) {
thread.setAutoArchiveDuration(sevenDaysInMinutes)
}
}
});
In plain English: whenever we see a new thread get created, we join the thread, send the <placeholder> message, and then immediately edit that message into @Keep. Then, we set the auto archive duration of the thread to 7 days if it wasn't already higher than that.
The Manual Archive
const archiveThreads = new CronJob('*/15 * * * *', async function() {
const guild = await client.guilds.fetch(GUILD)
const channels = await guild.channels.fetch()
const archiveThreshold = weekdaysBefore(moment(), 4)
channels
.filter(channel => channel.isText() && channel.name != "keep-github" && channel.viewable)
.forEach(async channel => {
const threads = await channel.threads.fetch()
threads.threads.forEach(async thread => {
const messages = await thread.messages.fetch({limit: 1})
if (moment(messages.first().createdTimestamp).isBefore(archiveThreshold)) {
const row = new MessageActionRow()
.addComponents(
new MessageButton()
.setCustomId('archive-thread')
.setLabel('Archive The Thread')
.setStyle('DANGER'),
);
thread.send({
content: `<@${thread.ownerId}>, it's been a bit since this thread has seen activity. Ready to archive it?`,
components: [row]
})
}
})
})
})
archiveThreads.start();
client.on('interactionCreate', async interaction => {
if (!interaction.isButton() || !interaction.customId === 'archive-thread') return;
const guild = await client.guilds.fetch(interaction.guildId)
const channel = await guild.channels.fetch(interaction.channelId)
await interaction.reply("Done!")
channel.setArchived(true)
});
This one is more complex. Ignoring the cron stuff and coming back to that later, first we dive into every thread, fetch the latest message, and see if its more than 4 weekdays old. If it is, we tag the author of the thread with a message that looks like this:
If they press the button, the thread is archived (and that's handled in the client.on('interactionCreate') handler). If they don't, this is activity in the thread that stops it from archiving, so the only way it'll ever archive is manually.
Finally, this process is run every 15 minutes via a cron schedule. You can decipher the arcane glyphs of cron script using a tool like https://crontab.guru/, so */15 * * * *, means "every 15th minute of every hour of every day of every month of every weekday" or "every 15 minutes".
Looking Forward
There are still things that bug me. Complex conversations still feel too flat, especially after using platforms like hackernews or reddit, where individual ideas can get their own scope and context (an even better version of threading!). Discord still has outages, and it's still a closed-source, centralized service where we don't own the data. It has a confusing monetization model, but has a bazillion users and it's snappy and polished as heck.
I don't think discord will be the thing forever, and for when it isn't any more and DeFi is ready to move, I hope the thing we move to is able to check my above boxes and add:
- End to End Encrypted
- Open Source
- Web3 friendly (sign in with ethereum, etc)