People sometimes ask what team building looks like in a remote company. The honest answer is: pretty normal. We talk, we ship things, we celebrate the small wins, and sometimes we stare into code that has been left cooking for a little too long.
Why are we doing this? Because we've decided to do so during our Sociocracy 3.0 meetings. You can read the rules in our handbook. Rock and Stone! . . . I mean Code and Test!
]]>When you accept GPU rentals from anyone willing to provide a suitable card - as we do - you have to be very careful. Sometimes, it can feel like shopping for a used car: if you don't have a reliable way to verify your purchase, you risk overpaying or getting scammed.
Well, not anymore. We couldn't accept this level of uncertainty, so we employed a 100% deterministic LLM to outsmart GPU scammers once and for all. Read on to discover how an ingenious idea and a sprinkle of Python magic helped us find a bulletproof method and reduce the GPUs needed for reliable verification by 240x.
Why test GPUs at all? At Reef Technologies, we’re creating a platform to sell GPU computing time from anyone online. Delivering high-quality service is crucial. So, we had to eliminate any uncertainty in GPU verification.
One way to test whether appropriate hardware is provided is to run something on it and verify the result. LLMs are popular, so let’s run an open-source one. Since they require a lot of VRAM, choosing a model with the right number of parameters allows us to ensure that at least X GB of VRAM is available. But how do we do that?
We started by taking a sample input. Then, we calculated the answer ourselves. Next, we compared the results and timing with the GPUs we needed to verify.
Imagine you’re a malicious provider with 100 nodes (each providing some GPU power), one node is really powerful, while the others are garbage. You receive the same test request on all 100 nodes. Cheating is simple: you calculate a result on the strongest node. Then, share the answer with your other nodes, and they will act as if they did the calculation themselves.
One may say, "Send different tasks to each node!"
But now our poor validator has to know answers to all the questions, i.e., perform as many calculations as there are nodes in the system. That’s a lot of work!
Also, aside from the problems above, let’s remember that we’re in the LLM era and we want to run LLMs on the GPUs, thus the validation task will likely not be just “2+2=?” but rather some text input completion using a huge LLM - so that we probe nodes on real-life tasks. And LLMs naturally introduce randomness - same input may produce different output - making direct comparison of validators’ and nodes’ answers unreliable.
So we’re in a bit of trouble here.
To solve the problem and make sure hardware differences could no longer be hidden, we decided to enforce strict determinism in our LLM testing. (Shame you can't do this when shopping for a car...)
So, how did we do it in practice?
Usually, libraries provide ways to ensure determinism. For example, Python’s built-in random module has a random.seed(123) function. For PyTorch, there is torch.use_deterministic_algorithms(True). Yes, this can result in performance degradation (why? slower, less-optimized but deterministic algorithms may be used, or, for example, some operations have no deterministic implementations on the GPU and thus fall back to CPU). But determinism was crucial to our approach, so it was definitely worth it.
In some cases (such as A100 GPUs), we also used float32 to guarantee stable outputs. For GPUs like the A6000, “auto” precision often suffices to get reproducible results.
Our setup_model function would set up a vllm.LLM instance with carefully chosen parameters: batch size, maximum token length, tensor parallelism, and enforced data types.
We disabled PyTorch’s cudnn.benchmark to keep execution paths consistent across runs.
This overarching setup ensures that the same prompt always yields the same output on the same GPU configuration as long as the batch size does not change (more on that in a moment).
We started by creating multiple large “prompt batches,” each sized to push a GPU to its capacity (e.g., 240 prompts). We chose this particular batch size to optimize memory usage, throughput, and guaranteed determinism on our reference GPUs.
It was at this stage that we made a critical discovery: a single prompt would result in different answers if we changed the batch size…
…but remained the same while the batch size was kept the same, even if mixed with different prompts!
Why is that? Changing the batch size can change the internal execution path. This leads to small numerical differences or different scheduling.
This is important because we can “mix” prompts from different original batches into one test batch, as long as the new batch has the same total size.
So we can answer 240 prompts in one go. Any GPU can. That means we have to answer 240 x NUMBER_OF_NODES prompts to validate them? If we send only a single prompt to each node, malicious providers can pretend to have 240 nodes using a single GPU. Or maybe we can use this fact to our advantage? Can we?
Let’s find answers for a batch of 240 prompts on trusted hardware, and then… split them up and hide them among “placeholder” prompts we don’t have answers to! So one answered batch yields 240 batches of:
But the tested nodes will not know which of the 240 they receive is the “real one”! They have to answer all of them (in a single go, that’s GPU vectori-batchi-black-hole-magification working). And we’ll only check the one answer. Only the one. Demotivating? Well, you’re a computer, deal with it. And if the answer doesn’t match the expected one, then:
This mixed batch approach allowed us to scale testing while minimizing compute costs. One single pass per GPU is enough to check whether it produces the correct, deterministic outputs and to measure its performance. We no longer have to re-run every original batch, which cuts our testing compute costs significantly.
Result? No more cheating!
Each prompt from the original batches must create the same output in the test batch. This means they need to match in size and seed. If there’s any difference, it could signal problems with the GPU or the provider’s processes. We can now easily verify all the hardware we supply to our users, without needing a car mechanic to come along!
Could this method be successfully applied to other computational problems? Absolutely. Feel free to experiment (and let us know)!
The main idea is to hide a test case among many others. The system must compute all of them since it can't tell which will be used for verification.
This method works well when creating test cases is cheap, but computing results is costly. It doesn’t have to depend on LLMs! For example, you can use prime factorization. Generating a number is easy, but factorizing it takes a lot of computation.
If you’d like to discuss our approach further, feel free to reach out!
]]>You may be wondering what a virtual assistant actually does, and how they could be of service to you in particular. Delegating your own duties to someone else probably makes you think of a large company’s CEO. After all, it is only at this stage of your career that you might need help arranging meetings and answering phone calls. Right? Well, it turns out that view of reality is quite outdated. The services of a virtual assistant can also make life much easier for programmers like you.
Think about the influential guy that you follow on social media. You read his articles, browse photos from his conferences, listen to an interview he gave for an industry-leading portal, and ask yourself… How does he make time for all these things? The answer is simple: there is another player that makes their success possible, and sometimes even a whole team. Virtual assistants make room in the client’s schedule through tireless behind-the-scenes work – so that he can focus on what’s really important.
So who is a VA, and what exactly does that person do? On the internet, you’ll find numerous definitions, often surprisingly different from each other. However, you can interpret this inconsistency as a good sign. The range of services offered by virtual assistants is so broad, with the assistants themselves so versatile and flexible that, in the end, everything depends on the client.
A virtual assistant is simply a remote helper who takes over some of your duties – especially those that will not suffer even if you don’t take care of them in person. Thanks to VA’s work, you gain extra time to focus on what you really want to do.
Virtual assistants do not limit themselves to typical business services. Increasingly so, they also support clients in less formal or even private, everyday matters. An experienced, resourceful assistant will take on some of the day-to-day responsibilities that right now are still yours to worry about.
During work, you will be able to fully focus on programming, and when you’re done with programming, you’ll spend time doing what you enjoy – and not dealing with commitments. If you want to delegate a task to an assistant, just type a few words on the keyboard. It’s really enough! Talking to virtual assistants works just like chatting with your colleagues. When you want someone to review your code, you send him a pull request link on Slack – and when you need your VA to help out, all you need is a brief description of your request, for instance:
“Hi, I’d like to eat pljeskavica today.” That is enough – the assistant will arrange everything for you. He will check which of the nearby Serbian restaurants has the best reviews, and then he will order a dish for you with home delivery or book a table and let you know where to find a parking space. There you go.
Everyone needs sufficient time for fun and leisure, that is what work-life balance is all about. It may sound like a corny slogan, but at Reef Technologies, we take it seriously. Our clients pay us for top quality code – and we realize only well-rested, satisfied employees can deliver that.
The help of a trusted assistant will allow you to stop worrying about everyday errands. The main objective of a VA can be put very simply: their services are there to make your life easier and more pleasant.
Everyday life consists of thousands of small things that require your time and energy – you may not even realize how many of them there are. Thanks to your assistant, you will be able to devote yourself to personal relationships, follow a wisely-planned path of self-development, and enjoy your life instead of coordinating tasks and errands. We prepared some specific examples of situations when a VA could provide you with the necessary support.
Your virtual assistant will:
Your virtual assistant will:
Your virtual assistant will:
Your virtual assistant will:
…it turns out you really do have a lot on your mind!
We’d wager that until now you’d never considered all those things as actual obligations. That’s because it was the way you’d lived all along. Somewhat overwhelming. Trust us, though, it is much easier to sleep soundly at night and lead a satisfying life when you can rest assured that a real professional is always on top of all your commitments. As a Reef Technologies developer, you’ll find out what it’s like for yourself.
]]>Despite several standard techniques dominating the IT recruitment industry, Reef Technologies went against the grain and developed their own unique approach. I asked the CEO, Paweł Polewicz, to explain why he felt the need to innovate, describe the recruitment process that he adopted, and tell me about its positive impact on the company.
Agnieszka: There are quite a few common recruitment methods to choose from, and most software houses simply follow one of them when looking for new developers. What makes Reef Technologies different?
Paweł: In the past, we tried using some standard techniques, but the results were not satisfactory – mainly because our offering is highly specific. We have a narrow, exclusive focus on backend solutions written in Python, usually web apps. A cookie-cutter approach wouldn’t work here – before we hire a developer, we need to make sure that they’re not just great at programming. They need to be great at the exact type of programming that we do here.
A: Before we start listing the interview stages, I would like to ask one question: why? Why does Reef Technologies have such a strong focus on the quality of the recruitment process?
P: Our Senior Python Developer position is rather attractive – it comes with a high hourly rate, a set of benefits, and the opportunity to participate in genuinely interesting projects. On top of that, we also work 100% remotely. As you can imagine, numerous programmers find that interesting and decide to apply.
Online testing is the best method we have found that allows us to handle large groups of candidates and select the very best while also making sure that our developers don’t have to spend excessive time supporting the recruitment process.
A: Wait, let me get this straight – so when it comes to filtering candidates, face-to-face conversations, and reading through CVs are not the best practice?
P: Not at all. Those methods favor the people who perform well in stressful face-to-face conversations or the people who know a good CV designer.
A: So… does that mean that after a candidate is accepted and becomes a staff member, they won’t need to spend their time reviewing CVs?
P: The last time an engineer at Reef Technologies saw a candidate’s CV was thousands of applicants ago.
A: How about other types of involvement in the recruitment process? Do your developers have to spend a significant amount of time verifying test solutions?
P: We wouldn’t want that, which is precisely why we have automated the process so heavily. These days, bots do most of the heavy lifting. We only have to engage with a tiny fraction of the candidates at the final stage of the process. Our approach doesn’t discriminate. It just tests what truly matters: the candidate’s skills.
A: Please give us a quick overview of how the interview process for the Senior Python Developer position works right now at Reef Technologies. What are the stages, and how much time can a candidate expect to spend in the process?
P: There are three stages: (1) A basic coding test that takes between 10 and 40 minutes. (2) A difficult debugging task, I’d say it takes around 2 hours to complete, but it varies significantly between candidates. And (3) a mini-project where the variance is even larger, as there are almost no limits to how little or how much time it can take. Good candidates are generally pretty fast, though.
Unfortunately, I cannot be very specific because it would skew the results of the test. What I can say is that a senior engineer takes 40 minutes on average to verify a single solution to the final, third stage of the process. If too many candidates were to reach it, the reviewing would drag us away from our projects far too often.
In light of the growing impact of LLMs, which are increasingly helping candidates complete tasks, we had to adapt our recruitment process. As a result, the number of stages has increased.
The first stage (1) is a short automated scripting task. Next (2), there is a programming game in which you build a decision-making solution for handling resources and logistics. Then (3), there is a 30-minute video call with our recruiter to assess general communication skills.
After that (4), there are two programming tasks focused on designing technical solutions and system architecture. Finally (5), there is a conversation with one of our engineers to get to know each other better, discuss the candidate’s experience, and introduce the final task.
After successfully completing all stages, we invite candidates to a fully paid evaluation / trial period.
A: Say a developer does well enough according to your metrics. What’s next?
P: If an engineer delivers adequate solutions at all three stages, we invite them to work with us on a fully paid trial period to see if we are a good match. Depending on the candidate, it may take between three days and three months before we decide to work together in the long term. If both parties are satisfied, we simply stop calling the relationship a trial period.
A: What made you choose those particular testing methods?
P: We continuously optimize the recruitment process for two main reasons. We want to optimize the candidate experience, but we’re also making an effort to automate it more and more so that we can focus on engineering tasks.
A: What improvements have you made to the recruitment process over the years? What made you realize that the process needed to be improved?
P: So far, we’ve had seven real iterations and five or so “virtual” iterations, where we analyzed the data and disproved a hypothesis without actually putting a hundred or so candidates through it. I’ve delivered a conference talk about this.
A: Please share some interesting data about the candidates. What were you surprised to find out about the average candidate for a senior-level Python job?
P: A significant portion of the candidates that we reject early do not have enough skill, to say, revert a string. 39% of our candidates fail that kind of task.
A: Reef Technologies provides very specific services. It means that once in a while, you may encounter a top-level programmer whose skill set is not compatible with the company. How do you handle such situations?
P: It happens all the time, every day – after all, we are looking for a needle in a haystack. Sometimes we find a golden ring, but that is not what we are looking for. For our projects, we only need the needles.
A: What advice would you give to a rejected candidate who wants to reach a skill level that would allow him to join Reef Technologies in a year?
P: Our training materials are publicly available in our “Handbook” repository on Github, where you can find very impactful articles and videos, both technical and organizational. Start with that, but then consider working with someone smarter than you. For instance, you might join a Python backend open-source project that welcomes contributions.
]]>