완전히 새로 정리된 Unreal + ONNX 통합 코드 (객체 감지 + 지형 파악) Part1

오늘 공부해볼 내용은 Unreal + ONNX 통합 코드 (객체 감지 + 지형 파악)에 대해 알아볼 것이다. 저번 시간에 언리얼 엔진(UE)에서 그 모델을 불러와 실시간으로 추론하고 결과를 반영하기 위해 모델을 불러오고 그 결과를 추론을 통해 텍스처로 어떻게 불러오는지까지 알아보았다. 하지만 이는 본래 목적이던 언리얼 학습의 맛보기일 뿐이다. 

우리는 학습 모델이 객체 감지 + 지형 파악을 추가로 할수 있도록 두 파트에 나누어서 공부를 진행해볼 예정이다. 언리얼 엔진에서 캡처하는 형식이 이미지 뿐만이 아닌 깊이데이터와 물체의 정보(일반 학습에서 태그를 붙인것과 유사하다) 또한 가져와 학습을 진행하는 코드를 Part1에서, 이 코드에 대해 알아보는 Part2로 알아볼 것이다.

 

 

 

---

 

 

1. 언리얼 프로젝트 위치

C:\Users\godling\Documents\Unreal Projects\UnrealCpp

 

2. ThirdParty 폴더 준비 (프로젝트 루트에ThirdParty폴더 생성)

UnrealCpp/
  ThirdParty/
    onnxruntime/                <-- unzip onnxruntime-win-x64-gpu-1.22.1
      include/
      lib/
      bin/                      <-- onnxruntime.dll 등
    opencv/
      include/
      x64/vc16/lib/
      x64/vc16/bin/             <-- opencv_world*.dll

 

 

• • onnxruntime은 C++ prebuilt 배포판 사용 권장(또는 직접 빌드).
  • OpenCV는 Windows prebuilt (Visual Studio 바이너리) 사용 권장.

3. ONNX 모델 파일 위치

• 객체감지 모델(예: YOLO 계열 ONNX): C:/Users/godling/Documents/Unreal Projects/UnrealCpp/Model/detector.onnx
• Depth 모델(예: MiDaS-like): .../Model/depth.onnx

 

4. RenderTarget

• 에디터에서 TextureRenderTarget2D (예: 640×480 혹은 512×512) 생성 → Actor에 연결

 

Build.cs

using UnrealBuildTool;
using System.IO;

public class UnrealCpp : ModuleRules
{
    public UnrealCpp(ReadOnlyTargetRules Target) : base(Target)
    {
        PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;

        PublicDependencyModuleNames.AddRange(new string[] {
            "Core", "CoreUObject", "Engine", "RenderCore", "RHI", "UMG"
        });

        PrivateDependencyModuleNames.AddRange(new string[] {  });

        // 프로젝트 경로
        string ProjectRoot = Path.GetFullPath(Path.Combine(ModuleDirectory, "../../"));
        string ThirdPartyPath = Path.Combine(ProjectRoot, "ThirdParty");

        // === OpenCV ===
        string OpenCVPath = Path.Combine(ThirdPartyPath, "opencv");
        PublicIncludePaths.Add(Path.Combine(OpenCVPath, "include"));
        // lib 파일 실제 파일명으로 바꾸기.
        PublicAdditionalLibraries.Add(Path.Combine(OpenCVPath, "x64", "vc16", "lib", "opencv_world412.lib"));
        // Runtime DLL (패키징 시 포함)
        RuntimeDependencies.Add(Path.Combine(OpenCVPath, "x64", "vc16", "bin", "opencv_world412.dll"));

        // === ONNX Runtime ===
        string OnnxPath = Path.Combine(ThirdPartyPath, "onnxruntime");
        PublicIncludePaths.Add(Path.Combine(OnnxPath, "include"));
        PublicAdditionalLibraries.Add(Path.Combine(OnnxPath, "lib", "onnxruntime.lib"));
        RuntimeDependencies.Add(Path.Combine(OnnxPath, "bin", "onnxruntime.dll"));
    }
}

 

 

주의: opencv_world412.lib / opencv_world412.dll 등 파일 이름은 OpenCV 버전(4.12 등)에 따라 달라진다. ThirdParty에 실제 파일명을 확인해서 Build.cs를 수정해야 한다.

 

 

에디터에서 해야할것

1. Visual Studio에서 솔루션 열기 → 빌드
2. 액터 배치
   • 에디터에서 빈 Actor 생성 → Components에 SceneCaptureComponent2D 추가, 혹은 미리 만든 Actor에 UOnnxVisionComponent(BlueprintSpawnableComponent)를 추가
   • RenderTarget(TextureRenderTarget2D) 할당(Resolution: 640×480 )
3. 모델 경로 지정
   • 컴포넌트 Details 패널에서 DetectorOnnxPath 와 DepthOnnxPath 에 모델 경로 입력 (절대 경로 권장 또는 프로젝트 Content 폴더 하위 경로)
4. Play
   • BeginPlay에서 모델 로드 - Tick에서 캡처/추론 시작

Source/UnrealCpp/OnnxVisionComponent.h

// OnnxVisionComponent.h
#pragma once

#include "CoreMinimal.h"
#include "Components/ActorComponent.h"
#include "Engine/TextureRenderTarget2D.h"
#include "Runtime/Engine/Classes/Components/SceneCaptureComponent2D.h"

#include <onnxruntime_cxx_api.h>
#include <opencv2/opencv.hpp>

#include "OnnxVisionComponent.generated.h"

UCLASS( ClassGroup=(Custom), meta=(BlueprintSpawnableComponent) )
class UNREALCPP_API UOnnxVisionComponent : public UActorComponent
{
    GENERATED_BODY()

public:    
    UOnnxVisionComponent();

protected:
    virtual void BeginPlay() override;

public:    
    virtual void TickComponent(float DeltaTime, ELevelTick TickType, FActorComponentTickFunction* ThisTickFunction) override;

    // 에디터에서 할당
    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Vision")
    USceneCaptureComponent2D* SceneCapture = nullptr;

    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Vision")
    UTextureRenderTarget2D* RenderTarget = nullptr;

    // 모델 경로 (프로젝트 폴더 절대 경로 권장)
    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Vision")
    FString DetectorOnnxPath;

    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Vision")
    FString DepthOnnxPath;

    // 디버그 옵션
    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Vision")
    bool bDrawDebugBoxes = true;

    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Vision")
    bool bShowDepthOverlay = true;

private:
    // ONNX Runtime
    Ort::Env OrtEnv;
    Ort::SessionOptions SessionOptions;
    std::unique_ptr<Ort::Session> DetectorSession;
    std::unique_ptr<Ort::Session> DepthSession;
    Ort::MemoryInfo CpuMemoryInfo;

    // helper
    cv::Mat RenderTargetToCvMat(UTextureRenderTarget2D* RT);
    std::vector<float> PreprocessForDetector(const cv::Mat& Img, int inW, int inH);
    std::vector<float> PreprocessForDepth(const cv::Mat& Img, int inW, int inH);
    UTexture2D* CreateTextureFromMat(const cv::Mat& Img); // for depth visualization

    void RunDetector(const std::vector<float>& inputTensor, int inputW, int inputH);
    void RunDepth(const std::vector<float>& inputTensor, int inputW, int inputH);
};

 

Source/UnrealCpp/OnnxVisionComponent.cpp

// OnnxVisionComponent.cpp
#include "OnnxVisionComponent.h"
#include "Engine/World.h"
#include "Kismet/KismetRenderingLibrary.h"
#include "Rendering/TextureRenderTargetResource.h"
#include "Engine/Texture2D.h"
#include "Async/Async.h"
#include "DrawDebugHelpers.h"
#include "Misc/Paths.h"

UOnnxVisionComponent::UOnnxVisionComponent()
    : OrtEnv(ORT_LOGGING_LEVEL_WARNING, "UE_OnnxVision"),
      CpuMemoryInfo(Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault))
{
    PrimaryComponentTick.bCanEverTick = true;
    SessionOptions.SetIntraOpNumThreads(1);
    SessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);
}

void UOnnxVisionComponent::BeginPlay()
{
    Super::BeginPlay();

    // 모델 파일 있으면 세션 생성
    if (!DetectorOnnxPath.IsEmpty())
    {
        std::string detPath = TCHAR_TO_UTF8(*DetectorOnnxPath);
        DetectorSession = std::make_unique<Ort::Session>(OrtEnv, detPath.c_str(), SessionOptions);
        UE_LOG(LogTemp, Warning, TEXT("Detector ONNX loaded: %s"), *DetectorOnnxPath);
    }

    if (!DepthOnnxPath.IsEmpty())
    {
        std::string dPath = TCHAR_TO_UTF8(*DepthOnnxPath);
        DepthSession = std::make_unique<Ort::Session>(OrtEnv, dPath.c_str(), SessionOptions);
        UE_LOG(LogTemp, Warning, TEXT("Depth ONNX loaded: %s"), *DepthOnnxPath);
    }
}

void UOnnxVisionComponent::TickComponent(float DeltaTime, ELevelTick TickType, FActorComponentTickFunction* ThisTickFunction)
{
    Super::TickComponent(DeltaTime, TickType, ThisTickFunction);

    if (!RenderTarget || !SceneCapture) return;

    // 1) 캡처
    SceneCapture->TextureTarget = RenderTarget;
    SceneCapture->CaptureScene();

    // 2) RT -> cv::Mat
    cv::Mat Img = RenderTargetToCvMat(RenderTarget);
    if (Img.empty()) return;

    // 기본 전처리 사이즈 (모델에 맞춰 조정하자)
    const int DET_W = 640, DET_H = 640;   // detector input
    const int DEP_W = 256, DEP_H = 256;   // depth input

    // 3) Detector 전처리 및 비동기 추론
    if (DetectorSession)
    {
        std::vector<float> detInput = PreprocessForDetector(Img, DET_W, DET_H);
        // run async to avoid blocking game thread
        Async(EAsyncExecution::ThreadPool, [this, detInput = MoveTemp(detInput), DET_W, DET_H]() mutable {
            RunDetector(detInput, DET_W, DET_H);
        });
    }

    // 4) Depth 전처리 및 비동기 추론
    if (DepthSession)
    {
        std::vector<float> depthInput = PreprocessForDepth(Img, DEP_W, DEP_H);
        Async(EAsyncExecution::ThreadPool, [this, depthInput = MoveTemp(depthInput), DEP_W, DEP_H]() mutable {
            RunDepth(depthInput, DEP_W, DEP_H);
        });
    }
}

// -------------------- helpers --------------------

cv::Mat UOnnxVisionComponent::RenderTargetToCvMat(UTextureRenderTarget2D* RT)
{
    if (!RT) return cv::Mat();

    FTextureRenderTargetResource* RTResource = RT->GameThread_GetRenderTargetResource();
    TArray<FColor> Bitmap;
    RTResource->ReadPixels(Bitmap);

    int Width = RT->SizeX;
    int Height = RT->SizeY;
    if (Bitmap.Num() != Width * Height) return cv::Mat();

    // Create cv::Mat from FColor array (BGRA)
    cv::Mat MatBGRA(Height, Width, CV_8UC4, Bitmap.GetData());
    cv::Mat MatRGB;
    cv::cvtColor(MatBGRA, MatRGB, cv::COLOR_BGRA2RGB);
    return MatRGB.clone(); // clone to ensure continuous memory
}

std::vector<float> UOnnxVisionComponent::PreprocessForDetector(const cv::Mat& Img, int inW, int inH)
{
    cv::Mat resized;
    cv::resize(Img, resized, cv::Size(inW, inH));
    cv::Mat floatImg;
    resized.convertTo(floatImg, CV_32F, 1.0f/255.0f);

    // YOLO 스타일: 1xCxHxW, channel order RGB, no normalization here or apply mean/std if needed
    int C = 3;
    std::vector<float> tensor(inW * inH * C);
    // HWC -> CHW
    for (int y=0; y<inH; ++y)
    {
        for (int x=0; x<inW; ++x)
        {
            cv::Vec3f pix = floatImg.at<cv::Vec3f>(y,x);
            for (int c=0; c<3; ++c)
            {
                int idx = c * (inW*inH) + y * inW + x;
                tensor[idx] = pix[c];
            }
        }
    }
    return tensor;
}

std::vector<float> UOnnxVisionComponent::PreprocessForDepth(const cv::Mat& Img, int inW, int inH)
{
    cv::Mat resized;
    cv::resize(Img, resized, cv::Size(inW, inH));
    cv::Mat floatImg;
    resized.convertTo(floatImg, CV_32F, 1.0f/255.0f);

    // use mean/std if depth model needs (modify as per your model)
    int C = 3;
    std::vector<float> tensor(inW * inH * C);
    for (int y=0; y<inH; ++y)
    {
        for (int x=0; x<inW; ++x)
        {
            cv::Vec3f pix = floatImg.at<cv::Vec3f>(y,x);
            for (int c=0; c<3; ++c)
            {
                int idx = c * (inW*inH) + y * inW + x;
                tensor[idx] = pix[c];
            }
        }
    }
    return tensor;
}

UTexture2D* UOnnxVisionComponent::CreateTextureFromMat(const cv::Mat& Img)
{
    if (Img.empty()) return nullptr;
    int W = Img.cols, H = Img.rows;

    // Ensure 3-channel BGR -> convert to BGRA
    cv::Mat BGRA;
    if (Img.channels() == 1)
        cv::cvtColor(Img, BGRA, cv::COLOR_GRAY2BGRA);
    else if (Img.channels() == 3)
        cv::cvtColor(Img, BGRA, cv::COLOR_BGR2BGRA);
    else if (Img.channels() == 4)
        BGRA = Img.clone();
    else
        return nullptr;

    // Create transient texture
    UTexture2D* Tex = UTexture2D::CreateTransient(W, H, PF_B8G8R8A8);
    if (!Tex) return nullptr;

    Tex->AddToRoot();
    Tex->UpdateResource();

    // Lock and copy
    FTexture2DMipMap& Mip = Tex->PlatformData->Mips[0];
    void* Data = Mip.BulkData.Lock(LOCK_READ_WRITE);
    FMemory::Memcpy(Data, BGRA.data, W * H * 4);
    Mip.BulkData.Unlock();
    Tex->UpdateResource();

    return Tex;
}

// -------------------- Inference runners --------------------

void UOnnxVisionComponent::RunDetector(const std::vector<float>& inputTensor, int inputW, int inputH)
{
    if (!DetectorSession) return;

    // Input shape: {1, C, H, W} — adjust if your model uses different layout
    std::array<int64_t,4> inputShape = {1, 3, inputH, inputW};
    size_t inputSize = inputTensor.size();

    // Create input tensor
    Ort::Value input = Ort::Value::CreateTensor<float>(CpuMemoryInfo, const_cast<float*>(inputTensor.data()), inputSize, inputShape.data(), inputShape.size());

    // input/output names
    Ort::AllocatorWithDefaultOptions allocator;
    char* inName = DetectorSession->GetInputNameAllocated(0, allocator).get();
    char* outName = DetectorSession->GetOutputNameAllocated(0, allocator).get();
    const char* inputNames[] = { inName };
    const char* outputNames[] = { outName };

    // Run
    auto outputs = DetectorSession->Run(Ort::RunOptions{nullptr}, inputNames, &input, 1, outputNames, 1);

    // NOTE: parsing depends on model output format!
    // Here we assume detector outputs Nx6 array: [x1, y1, x2, y2, score, class]
    float* outData = outputs.front().GetTensorMutableData<float>();
    auto outTypeInfo = outputs.front().GetTensorTypeAndShapeInfo();
    std::vector<int64_t> outShape = outTypeInfo.GetShape();
    size_t N = 1;
    for (auto d : outShape) N *= d;
    // If shape is (num,6) or (1, num, 6), handle accordingly. We attempt generic handling:
    int64_t rows = (outShape.size() >= 2) ? outShape[outShape.size()-2] : (int64_t)(N/6);
    int64_t cols = (outShape.size() >= 1) ? outShape.back() : 6;

    // Collect detections
    std::vector< TTuple<float,float,float,float,int,float> > dets; // placeholder: not compile-time type; we'll draw inline below

    // For simplicity, interpret as flat (num x 6)
    int numBoxes = N / cols;
    struct Box { float x1,y1,x2,y2,score; int cls; };
    std::vector<Box> boxes;
    for (int i=0;i<numBoxes;i++)
    {
        int base = i * cols;
        if (cols < 6) continue;
        float x1 = outData[base + 0];
        float y1 = outData[base + 1];
        float x2 = outData[base + 2];
        float y2 = outData[base + 3];
        float score = outData[base + 4];
        int cls = (int)outData[base + 5];

        if (score < 0.3f) continue; // threshold

        // Convert detector coords (assumed to be on resized input) to render target coords if needed
        // Here we assume detector used same dims as the resized input; to map to world we need scale factors
        boxes.push_back({x1,y1,x2,y2,score,cls});
    }

    // Draw boxes on game thread
    AsyncTask(ENamedThreads::GameThread, [this, boxes]() {
        UWorld* World = GetWorld();
        if (!World) return;

        for (const auto& b : boxes)
        {
            // Convert 2D screen coords to world debug boxes: approximate using camera location/direction, or draw 2D via UCanvas
            // For simplicity, draw 2D rectangles via DrawDebugBox in world at actor location (visual approximation)
            FVector Center = GetOwner()->GetActorLocation() + FVector(0,0,100); // offset upwards
            FVector Extent = FVector(FMath::Abs(b.x2 - b.x1)*0.01f, FMath::Abs(b.y2 - b.y1)*0.01f, 50.0f);
            DrawDebugBox(World, Center, Extent, FColor::Red, false, 0.5f);
            // Show label as on-screen message
            if (GEngine)
            {
                FString Msg = FString::Printf(TEXT("Det class %d score %.2f"), b.cls, b.score);
                GEngine->AddOnScreenDebugMessage(-1, 1.5f, FColor::Yellow, Msg);
            }
        }
    });
}

void UOnnxVisionComponent::RunDepth(const std::vector<float>& inputTensor, int inputW, int inputH)
{
    if (!DepthSession) return;

    std::array<int64_t,4> inputShape = {1, 3, inputH, inputW};
    size_t inputSize = inputTensor.size();

    Ort::Value input = Ort::Value::CreateTensor<float>(CpuMemoryInfo, const_cast<float*>(inputTensor.data()), inputSize, inputShape.data(), inputShape.size());

    Ort::AllocatorWithDefaultOptions allocator;
    char* inName = DepthSession->GetInputNameAllocated(0, allocator).get();
    char* outName = DepthSession->GetOutputNameAllocated(0, allocator).get();
    const char* inputNames[] = { inName };
    const char* outputNames[] = { outName };

    auto outputs = DepthSession->Run(Ort::RunOptions{nullptr}, inputNames, &input, 1, outputNames, 1);

    // Assume output is [1,1,H,W] or [1,H,W], float depth values
    float* outData = outputs.front().GetTensorMutableData<float>();
    auto outInfo = outputs.front().GetTensorTypeAndShapeInfo();
    std::vector<int64_t> shape = outInfo.GetShape();

    int H = (int)shape.back();
    int W = (int)shape[shape.size()-2];
    if (shape.size() == 4) { H = (int)shape[2]; W = (int)shape[3]; }
    // build cv::Mat grayscale float
    cv::Mat depthMat(H, W, CV_32F);
    memcpy(depthMat.data, outData, sizeof(float) * H * W);

    // normalize for visualization
    double minv, maxv;
    cv::minMaxLoc(depthMat, &minv, &maxv);
    cv::Mat vis;
    depthMat.convertTo(vis, CV_8U, 255.0 / (maxv - minv + 1e-6), -minv * 255.0 / (maxv - minv + 1e-6));
    cv::applyColorMap(vis, vis, cv::COLORMAP_JET);

    // send to game thread to show as texture overlay
    AsyncTask(ENamedThreads::GameThread, [this, visMat = vis.clone()]() mutable {
        UTexture2D* Tex = CreateTextureFromMat(visMat);
        if (!Tex) return;
        // Option A: set as material parameter on an actor's mesh
        // Option B: draw via UMG/Widget - here we use on-screen debug print and show that texture not trivial in code snippet
        GEngine->AddOnScreenDebugMessage(-1, 1.5f, FColor::Cyan, TEXT("Depth visualized"));
        // You can store Tex in a UPROPERTY to display in UI/Material
    });
}