Temper Bench v2
Первая версия собственного бенчмарка, на которой оценивали Temper 0.5B, со временем перестала отвечать тому, что мы от неё ждали. В ней было 100 задач трёх уровней: 50 тривиальных, 30 простых и 20 средних. Пересечений с обучающим датасетом и утечек в ней не было, но тексты задач готовили те же модели DeepSeek, что генерировали обучающие данные, поэтому бенчмарк совпадал с датасетом по стилю. К тому же большую часть составляли короткие алгоритмические упражнения вроде проверки числа на простоту, факториала, НОД или FizzBuzz. Они хорошо проверяют знание синтаксиса, но мало говорят о том, как модель справляется с реальным кодом, а именно это для нас было главным.
Вторую версию мы писали так, чтобы она стала показателем прикладной работы. Вместо алгоритмических упражнений в ней небольшие самостоятельные задачи, похожие на типичные части реальных проектов, например расчёт стоимости доставки, разбор конфига, разбор HTTP-запроса или кэш, с которым одновременно работают несколько потоков. На каждую задачу приходится от пяти тестов, в них обязательно есть пустой ввод и крайние случаи. Бенчмарк получился непростым даже для крупных моделей: Qwen2.5-Coder-7B с первой попытки решает 46% базовых задач, 22% средних и только 10% продвинутых.
Примеры задач
Write normalize_phone(raw: &str) -> Option<String> for US phone numbers. Ignore everything that isn't a digit. If there are 10 digits, or 11 digits starting with 1, format the last 10 as (555) 123-4567. Anything else gives None. Reply with a single file containing the solution.
Скрытые тесты
#[cfg(test)]
mod tests {
use super::*;
fn some(s: &str) -> Option<String> {
Some(s.to_string())
}
#[test]
fn plain_ten_digits() {
assert_eq!(normalize_phone("5551234567"), some("(555) 123-4567"));
}
#[test]
fn various_separators() {
assert_eq!(normalize_phone("555-123-4567"), some("(555) 123-4567"));
assert_eq!(normalize_phone("(555) 123 4567"), some("(555) 123-4567"));
assert_eq!(normalize_phone("555.123.4567"), some("(555) 123-4567"));
}
#[test]
fn country_code() {
assert_eq!(normalize_phone("+1 555 123 4567"), some("(555) 123-4567"));
assert_eq!(normalize_phone("15551234567"), some("(555) 123-4567"));
}
#[test]
fn eleven_digits_without_one() {
assert_eq!(normalize_phone("25551234567"), None);
}
#[test]
fn wrong_length() {
assert_eq!(normalize_phone("555123456"), None);
assert_eq!(normalize_phone("115551234567"), None);
assert_eq!(normalize_phone(""), None);
assert_eq!(normalize_phone("call me"), None);
}
#[test]
fn letters_ignored() {
assert_eq!(normalize_phone("tel: 555 123 4567 (home)"), some("(555) 123-4567"));
}
}Эталонное решение
pub fn normalize_phone(raw: &str) -> Option<String> {
let digits: String = raw.chars().filter(|c| c.is_ascii_digit()).collect();
let digits = match digits.len() {
10 => digits.as_str(),
11 if digits.starts_with('1') => &digits[1..],
_ => return None,
};
Some(format!("({}) {}-{}", &digits[..3], &digits[3..6], &digits[6..]))
}Implement a loader for a small key = value configuration format, with errors reported through Result and the ? operator. Use this API:
pub fn parse_config(text: &str) -> Result<Config, ConfigError>;Config is a struct with the public fields name: String, retries: u32 and timeout_ms: u64. ConfigError is an enum with these variants: MissingKey(&'static str), BadLine(usize) and BadNumber { key: String, source: ParseIntError }, where ParseIntError is std::num::ParseIntError. Both types derive Debug and PartialEq. Rules: - Each line is trimmed. Blank lines and lines that start with # are ignored. - Every other line must contain =. Key and value are trimmed. A line without = gives ConfigError::BadLine with its 1-based number in the text, counting blank and comment lines. - The keys retries and timeout_ms are parsed as numbers of their types. A value that does not parse gives BadNumber with the key. The key name is taken as text. - Unknown keys are ignored. When a key appears more than once, the last value wins. - After all lines are read, a missing name gives MissingKey("name"), then a missing retries gives MissingKey("retries"), then a missing timeout_ms gives MissingKey("timeout_ms"). Check them in this order. Reply with a single file containing the solution.
Скрытые тесты
#[cfg(test)]
mod tests {
use super::*;
const FULL: &str = "name = api\nretries = 3\ntimeout_ms = 1500\n";
#[test]
fn parses_full_config() {
assert_eq!(
parse_config(FULL),
Ok(Config {
name: "api".to_string(),
retries: 3,
timeout_ms: 1500,
})
);
}
#[test]
fn skips_blank_and_comment_lines() {
let text = "# header\n\n name=api \nretries=0\n# note\ntimeout_ms=5\n";
let config = parse_config(text).unwrap();
assert_eq!(config.name, "api");
assert_eq!(config.retries, 0);
}
#[test]
fn line_without_equals_reports_its_line() {
let text = "name = api\nretries 3\ntimeout_ms = 1\n";
assert_eq!(parse_config(text), Err(ConfigError::BadLine(2)));
}
#[test]
fn bad_number_names_the_key() {
let text = "name = api\nretries = x\ntimeout_ms = 1\n";
assert!(matches!(
parse_config(text),
Err(ConfigError::BadNumber { ref key, .. }) if key == "retries"
));
}
#[test]
fn negative_retries_is_bad_number() {
let text = "name = api\nretries = -1\ntimeout_ms = 1\n";
assert!(matches!(parse_config(text), Err(ConfigError::BadNumber { .. })));
}
#[test]
fn missing_keys_checked_in_order() {
assert_eq!(parse_config(""), Err(ConfigError::MissingKey("name")));
assert_eq!(
parse_config("name = a\ntimeout_ms = 1\n"),
Err(ConfigError::MissingKey("retries"))
);
assert_eq!(
parse_config("name = a\nretries = 1\n"),
Err(ConfigError::MissingKey("timeout_ms"))
);
}
#[test]
fn last_value_wins_and_unknown_keys_ignored() {
let text = "name = old\nname = new\nretries = 1\ntimeout_ms = 2\ncolor = blue\n";
let config = parse_config(text).unwrap();
assert_eq!(config.name, "new");
}
}Эталонное решение
use std::num::ParseIntError;
use std::str::FromStr;
#[derive(Debug, PartialEq)]
pub struct Config {
pub name: String,
pub retries: u32,
pub timeout_ms: u64,
}
#[derive(Debug, PartialEq)]
pub enum ConfigError {
MissingKey(&'static str),
BadLine(usize),
BadNumber { key: String, source: ParseIntError },
}
fn number<T: FromStr<Err = ParseIntError>>(key: &str, value: &str) -> Result<T, ConfigError> {
value
.parse::<T>()
.map_err(|source| ConfigError::BadNumber {
key: key.to_string(),
source,
})
}
pub fn parse_config(text: &str) -> Result<Config, ConfigError> {
let mut name = None;
let mut retries = None;
let mut timeout_ms = None;
for (index, raw) in text.lines().enumerate() {
let line = raw.trim();
if line.is_empty() || line.starts_with('#') {
continue;
}
let (key, value) = line.split_once('=').ok_or(ConfigError::BadLine(index + 1))?;
let (key, value) = (key.trim(), value.trim());
match key {
"name" => name = Some(value.to_string()),
"retries" => retries = Some(number::<u32>(key, value)?),
"timeout_ms" => timeout_ms = Some(number::<u64>(key, value)?),
_ => {}
}
}
Ok(Config {
name: name.ok_or(ConfigError::MissingKey("name"))?,
retries: retries.ok_or(ConfigError::MissingKey("retries"))?,
timeout_ms: timeout_ms.ok_or(ConfigError::MissingKey("timeout_ms"))?,
})
}Write a parser for plain HTTP/1.x requests in Rust. The method, path, header names and values and the body all borrow from the input string. Use only std. Use this API:
#[derive(Debug, PartialEq)]
pub struct Request<'a> {
pub method: &'a str,
pub path: &'a str,
pub headers: Vec<(&'a str, &'a str)>,
pub body: &'a str,
}
#[derive(Debug, PartialEq)]
pub enum RequestError {
BadRequestLine,
BadVersion(String),
BadHeader(usize),
Truncated,
LengthMismatch { declared: usize, actual: usize },
}
pub fn parse_request<'a>(raw: &'a str) -> Result<Request<'a>, RequestError>;The input uses \n as the line separator, and no other handling of line endings is done. Lines are numbered from 1, and the request line is line 1. - The request line must have exactly three parts separated by single spaces: the method, the path and the version. A part that is empty, or a line with more or fewer than three parts, gives BadRequestLine. - The version must be HTTP/1.1 or HTTP/1.0. Any other version gives BadVersion with the version text. - The headers follow the request line, one per line, and they end at the first empty line. A header line must contain a :. The name is the text before the first :, and it must not be empty and must not contain a space. Otherwise the result is BadHeader with the line number. The value is the text after the first :, with the spaces at its start and end removed. - If the input ends before an empty line is found, the result is Truncated. - The body is everything after the empty line that ends the headers. It may be empty. - If a header named Content-Length (compared without regard to ASCII case) is present, only the first such header counts. Its value must be a decimal number, or the result is BadHeader with its line number. The number must equal the length of the body in bytes, or the result is LengthMismatch with the declared number and the actual length. Without that header, the body length is not checked. Reply with a single file containing the solution.
Скрытые тесты
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn get_with_headers_and_no_body() {
let raw = "GET /index.html HTTP/1.1\nHost: example.org\nAccept: text/html \n\n";
assert_eq!(
parse_request(raw),
Ok(Request {
method: "GET",
path: "/index.html",
headers: vec![("Host", "example.org"), ("Accept", "text/html")],
body: "",
})
);
}
#[test]
fn body_after_blank_line() {
let raw = "POST /x HTTP/1.0\n\nhello";
let req = parse_request(raw).unwrap();
assert_eq!(req.body, "hello");
assert!(req.headers.is_empty());
}
#[test]
fn content_length_must_match_body_bytes() {
let raw = "PUT /a HTTP/1.1\ncontent-length: 5\n\nhéllo";
assert_eq!(
parse_request(raw),
Err(RequestError::LengthMismatch { declared: 5, actual: 6 })
);
let ok = "PUT /a HTTP/1.1\nContent-Length: 6\n\nhéllo";
assert_eq!(parse_request(ok).unwrap().body, "héllo");
}
#[test]
fn first_content_length_header_counts() {
let raw = "PUT /a HTTP/1.1\nContent-Length: 2\nContent-Length: 9\n\nab";
assert_eq!(parse_request(raw).unwrap().body, "ab");
}
#[test]
fn content_length_that_is_not_a_number_is_bad_header() {
let raw = "PUT /a HTTP/1.1\nContent-Length: two\n\n";
assert_eq!(parse_request(raw), Err(RequestError::BadHeader(2)));
}
#[test]
fn request_line_needs_exactly_three_single_spaced_parts() {
assert_eq!(parse_request("GET /x\n\n"), Err(RequestError::BadRequestLine));
assert_eq!(parse_request("GET /x HTTP/1.1\n\n"), Err(RequestError::BadRequestLine));
assert_eq!(parse_request("GET /x HTTP/1.1 extra\n\n"), Err(RequestError::BadRequestLine));
}
#[test]
fn unsupported_version() {
assert_eq!(
parse_request("GET / HTTP/2.0\n\n"),
Err(RequestError::BadVersion("HTTP/2.0".to_string()))
);
}
#[test]
fn header_without_colon_reports_line() {
assert_eq!(
parse_request("GET / HTTP/1.1\nHost example\n\n"),
Err(RequestError::BadHeader(2))
);
}
#[test]
fn header_name_with_space_or_empty_name_is_bad() {
assert_eq!(
parse_request("GET / HTTP/1.1\nBad Name: x\n\n"),
Err(RequestError::BadHeader(2))
);
assert_eq!(
parse_request("GET / HTTP/1.1\n: x\n\n"),
Err(RequestError::BadHeader(2))
);
}
#[test]
fn missing_blank_line_is_truncated() {
assert_eq!(
parse_request("GET / HTTP/1.1\nHost: a"),
Err(RequestError::Truncated)
);
}
#[test]
fn fields_point_into_the_input() {
let raw = String::from("DELETE /item HTTP/1.1\n\n");
let req = parse_request(&raw).unwrap();
let start = raw.as_ptr() as usize;
assert_eq!(req.path.as_ptr() as usize - start, 7);
}
}Эталонное решение
#[derive(Debug, PartialEq)]
pub struct Request<'a> {
pub method: &'a str,
pub path: &'a str,
pub headers: Vec<(&'a str, &'a str)>,
pub body: &'a str,
}
#[derive(Debug, PartialEq)]
pub enum RequestError {
BadRequestLine,
BadVersion(String),
BadHeader(usize),
Truncated,
LengthMismatch { declared: usize, actual: usize },
}
pub fn parse_request<'a>(raw: &'a str) -> Result<Request<'a>, RequestError> {
let mut lines = raw.split('\n');
let request_line = lines.next().ok_or(RequestError::BadRequestLine)?;
let parts: Vec<&str> = request_line.split(' ').collect();
if parts.len() != 3 || parts.iter().any(|part| part.is_empty()) {
return Err(RequestError::BadRequestLine);
}
let (method, path, version) = (parts[0], parts[1], parts[2]);
if version != "HTTP/1.1" && version != "HTTP/1.0" {
return Err(RequestError::BadVersion(version.to_string()));
}
let mut headers = Vec::new();
let mut declared_length: Option<(usize, &str)> = None;
let mut consumed = request_line.len() + 1;
let mut found_blank = false;
for (index, line) in lines.enumerate() {
consumed += line.len() + 1;
if line.is_empty() {
found_blank = true;
break;
}
let number = index + 2;
let (name, value) = line.split_once(':').ok_or(RequestError::BadHeader(number))?;
if name.is_empty() || name.contains(' ') {
return Err(RequestError::BadHeader(number));
}
let value = value.trim_matches(' ');
if declared_length.is_none() && name.eq_ignore_ascii_case("Content-Length") {
declared_length = Some((number, value));
}
headers.push((name, value));
}
if !found_blank {
return Err(RequestError::Truncated);
}
let body = raw.get(consumed..).unwrap_or("");
if let Some((number, value)) = declared_length {
let declared: usize = value.parse().map_err(|_| RequestError::BadHeader(number))?;
if declared != body.len() {
return Err(RequestError::LengthMismatch {
declared,
actual: body.len(),
});
}
}
Ok(Request {
method,
path,
headers,
body,
})
}Ограничения
Мы старались сделать Temper Bench v2 честным и качественным. Задачи написаны с нуля, решения проверяют компилятор и скрытые тесты, бенчмарк проверен на утечки. Но мы не исследовательская лаборатория и раньше бенчмарки не создавали, поэтому ставить его в один ряд с HumanEval или MBPP, которые годами проверяло сообщество, пока рано. Мы сделали его, потому что для Rust почти нет готовых альтернатив, кроме переводов этих же наборов. Окончательный вывод о модели всё равно стоит делать на своих задачах, попробовав её в работе.