---
title: "Boosting Language Learning with AI: Building a Real-Time Voice Translation Assistant"
description: "Combine Azure Speech Recognition with Azure OpenAI ChatGPT to build a Python language-learning assistant that translates speech in real time and generates example sentences."
canonical_url: "https://blog.markkulab.net/en/post/ai-english-learning-assistant"
author: "Mark Ku"
author_url: "https://blog.markkulab.net/en/author/mark-ku"
site: "Mark Ku's Tech Notes"
date_published: "2024-05-21 01:01:35 +0800"
category: "AI"
tags: ["ai", "azure", "speech-recognition", "openai", "python", "language-learning"]
language: "en"
license: "CC BY 4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
attribution: "when reusing or quoting, credit the author and link back to the original"
---

# Boosting Language Learning with AI: Building a Real-Time Voice Translation Assistant

## Introduction
In a world where multilingual communication keeps growing in importance, picking up a foreign language has become a common goal. The catch is that language learning takes time, patience, and—most importantly—effective opportunities to practice. This post walks through how to combine Microsoft Azure Speech Recognition with OpenAI's ChatGPT to build a tool that translates and responds in real time, making it easier to learn and apply a new language.

## Prerequisites
Before getting started, you'll need:
* A Python environment to run the script.
* A Microsoft Azure subscription for the Speech service.
* An Azure OpenAI API key for generating ChatGPT responses.

## How It Works
At its core, the system pairs Azure's speech recognition with OpenAI's language understanding. The flow breaks down into the following steps:

1. Voice input: the system captures audio through the microphone.
2. Speech recognition: Azure's Speech service transcribes the audio into text.
3. Response generation: the recognized text is sent to ChatGPT, which produces three short example sentences.
4. Display: the responses are shown to the user in real time.

Code Implementation
Below is a simple Python script that shows how to wire up Azure Speech and the OpenAI ChatGPT model:

```
import os
import azure.cognitiveservices.speech as speechsdk
from langchain.prompts import ChatPromptTemplate
from langchain_openai import AzureChatOpenAI

# 設定OpenAI環境變數
os.environ["AZURE_OPENAI_ENDPOINT"] = "https://xxx.openai.azure.com/"
os.environ["AZURE_OPENAI_API_VERSION"] = "2024-02-15-preview"
os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"] = "gpt3-turbo"

# 初始化OpenAI模型
model = AzureChatOpenAI(
    openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
    azure_deployment=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"],
)

# 為OpenAI設定聊天模板
chat_template = ChatPromptTemplate.from_messages(
    [
        ("system", "你是一個國中英文老師.善於用簡單的單詞溝通."),
        ("human", "關於英文問題{question}, 給我怎麼回覆的三個句子，並給中文翻譯. 比較音節困難的單字額外給音標，音標放在單字後面"),
    ]
)

def get_responses(question):
    message = chat_template.format_messages(question=question)
    response = model.invoke(message)
    return response.content

def translate_speech_to_text_continuous():
    # 用於語音翻譯的Azure訂閱金鑰和服務區域
    speech_key = "<Your-Azure-Speech-Key>"
    service_region = "eastus"
    from_language = "en-US"
    to_language = "zh-Hant"

    # 創建語音翻譯配置和識別器
    translation_config = speechsdk.translation.SpeechTranslationConfig(
        subscription=speech_key, region=service_region,
        speech_recognition_language=from_language)
    translation_config.add_target_language(to_language)

    recognizer = speechsdk.translation.TranslationRecognizer(translation_config=translation_config)

    with open("translation_log.txt", "w", encoding="utf-8") as log_file:
        def recognized_handler(evt):
            result = evt.result
            if result.reason == speechsdk.ResultReason.TranslatedSpeech:
                recognized_text = f"Recognized: {result.text}\n"
                translated_text = f"Translated into Traditional Chinese: {result.translations['zh-Hant']}\n"
                print(recognized_text)
                print(translated_text)
                log_file.write(recognized_text)
                log_file.write(translated_text)

                # 如果識別的文字是問題，獲得簡單的例句回答
                responses = get_responses(result.text)
                print("例句:\n")
                print(responses)
                log_file.write(f"Responses: {responses}\n")

            elif result.reason == speechsdk.ResultReason.NoMatch:
                no_match_text = "No speech could be recognized\n"
                print(no_match_text)
                # log_file.write(no_match_text)
            elif result.reason == speechsdk.ResultReason.Canceled:
                cancellation_details = result.cancellation_details
                canceled_text = f"Speech Recognition canceled: {cancellation_details.reason}\n"
                # print(canceled_text)
                # log_file.write(canceled_text)
                if cancellation_details.reason == speechsdk.CancellationReason.Error:
                    error_text = f"Error details: {cancellation_details.error_details}\n"
                    print(error_text)
                    log_file.write(error_text)

        recognizer.recognized.connect(recognized_handler)

        # 開始連續識別
        recognizer.start_continuous_recognition()
        print("Listening...")

        try:
            # 保持程序運行直到按Enter鍵停止
            input("Press Enter to stop...\n")
        finally:
            recognizer.stop_continuous_recognition()

if __name__ == "__main__":
    translate_speech_to_text_continuous()
```
## Running the Script
1. When you run the Python script, the interaction unfolds as follows:
2. Startup prompt: once the script launches, the system shows "Please say something..." to invite the user to speak.
3. Voice input: speak into the microphone. The system captures whatever you say.
4. Recognition output: as soon as you stop speaking, Azure's Speech service transcribes the audio into text. This is usually very fast. The recognized text appears on screen, for example: "Recognized text: Hello, I'd like to know more about AI."
5. Response generation: the recognized text is forwarded to ChatGPT, which generates a response based on that input. This typically takes a few seconds.
![final-screen](https://blog.markkulab.net/content/markku/posts/ai-english-learning-assistant/images/final-screen.jpg)

---

## About this article and its author

Originally published on [Mark Ku's Tech Notes](https://blog.markkulab.net/en/post/ai-english-learning-assistant)

License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) — when reusing or quoting, credit the author and link back to the original

### About the author

**[Mark Ku](https://blog.markkulab.net/en/author/mark-ku)** — Software Solution Provider

- 10+ years senior software engineer, now an AI Builder
- Focused on large-platform architecture — North-American e-commerce, AI SaaS subscription billing
- Combining AI Agents and automation to build evolvable product foundations

### Free tools built by the author

All of these are free to use:

- [Free PDF Sign Tool](https://blog.markkulab.net/en/tools/pdf-sign): Online PDF sign tool — draw, type, or upload a signature, then drag, resize, and download. Everything runs in your browser; nothing is uploaded.
- [VS Code Refactory](https://blog.markkulab.net/en/tools/refactory): Refactory is a VS Code refactoring extension: 34 actions plus a 37-rule code-smell inspection layer with a Code Health dashboard, across 18 languages, backed by 534 tests. It learns your repo's conventions: where interfaces live, where DI is registered, whether 'use client' belongs. It ranks files by git churn × complexity so you know what to fix first, and hands any smell to the Claude Code already on your machine. Free to use, and your source never leaves your computer.
- [DB-Kit Database Manager](https://blog.markkulab.net/en/tools/db-kit): DB-Kit is a lightweight, cross-platform database manager built with Tauri + Rust + React. Manage MySQL, MariaDB, PostgreSQL, SQL Server, Oracle, SQLite, MongoDB, Redis, Kafka, Elasticsearch and RabbitMQ from one consistent interface: passwords encrypted in the OS keychain, SSH tunnels, full CRUD, a visual query builder, stacked multi-statement result sets, cross-connection data transfer and compare/sync, Excel / CSV import & export, visualized execution plans, ER diagrams, scheduled backups, SQL stress testing with p50–p99 latency percentiles, a 15-rule SQL review engine, Kafka message browsing with monitoring & alerts, a bilingual UI (Traditional Chinese / English), a built-in AI assistant (natural-language SQL, AI review and tuning advice) and the dbk CLI. Free and open source (MIT), with installers for Windows, macOS and Linux.
- [VS Code Super Mermaid](https://blog.markkulab.net/en/tools/super-mermaid): Super Mermaid is a VS Code extension for beautiful Mermaid diagrams out of the box: auto-colored live preview, mouse pan & zoom, high-res PNG / SVG export, 21 templates and multiple themes. Free and open source (MIT).
- [React Super Mermaid](https://blog.markkulab.net/en/tools/react-super-mermaid): react-super-mermaid is an open-source React component library: render beautiful Mermaid diagrams with a single <MermaidViewer>, with built-in colorful / sketch themes, pan & zoom, in-diagram search, and high-res SVG / PNG export. Lightweight, SSR-safe, fully typed. Free and open source (MIT).
- [Jira / Confluence Super Mermaid](https://blog.markkulab.net/en/tools/jira-super-mermaid): An Atlassian Forge app: write Mermaid syntax directly inside a Jira issue or a Confluence page and get flowcharts, sequence diagrams, state machines and Gantt charts. 11 diagram types, SVG / PNG export, light and dark themes, full CJK support. Runs on Atlassian: your diagrams live in your own site and the app calls no third-party service. Free, coming soon to the Atlassian Marketplace.
- [Mermaid Live Preview](https://blog.markkulab.net/en/tools/mermaid-preview): Write Mermaid in your browser, see it render instantly, and share the whole diagram as a single link. No sign-up, nothing uploaded to a server, and mermaid.live share links work as-is.
- [React Intl Phone Number](https://blog.markkulab.net/en/tools/react-intl-phone-number): react-intl-phone-number is an open-source React component: framework-agnostic and antd-free, with E.164 in/out, a searchable flag / country-code dropdown, configurable validation levels (strict / mobile-strict / loose), themeable CSS, and i18n — phone logic powered by google-libphonenumber. Lightweight and fully typed. Free and open source (MIT).
- [Uptime Kuma Cluster](https://blog.markkulab.net/en/tools/uptime-kuma-cluster): Turn single-node Uptime Kuma into a highly available cluster: OpenResty + Lua smart load balancing, shared MariaDB state, health checks and automatic failover, plus cluster-management REST APIs. One Docker Compose command to start. Free and open source (MIT).
- [Special Education](https://blog.markkulab.net/en/education): Learning materials crafted for special education students

### Daily podcasts

- [Mark's Tech Insights — Daily AI News](https://blog.markkulab.net/en/category/tech-news): Daily curated AI and tech trends. Catch the latest developments via audio summaries — covering AI applications, software architecture, DevOps, and engineering practice. — RSS: https://blog.markkulab.net/feed.xml
- [AI股市蝦聊](https://blog.markkulab.net/en/category/ai-stock-chat): Every trading day, an AI-analyzed take on the Taiwan stock market, delivered as a two-host conversation covering the session and the next-day outlook. — RSS: https://blog.markkulab.net/ai-stock-chat/feed.xml
- [開源好物週報](https://blog.markkulab.net/en/category/open-source-weekly): A weekly two-host pick of free open-source tools surfaced from real Hacker News, GitHub, and Reddit buzz — what pain they solve and the fastest way to get started. — RSS: https://blog.markkulab.net/open-source-weekly/feed.xml

### Newsletter

[Subscribe to the newsletter](https://blog.markkulab.net/en/subscribe) — Be the first to know about new posts. No spam, unsubscribe anytime.