LogoLogo
HomeExplorerGitHub
  • 🗂️Introduction
    • 🧐5Ws-1H about CESS
    • ✨Technical Highlight
    • 🔨Use Cases
    • 📜Whitepaper
    • 💬Contact & Social Media
  • ⛏️CESS Nodes
    • Consensus Nodes
      • Running a Consensus Node
      • Reward Mechanism
    • Storage Nodes
      • Running a Storage Node
      • Running MultiNodes
      • Node Troubleshooting
      • Storage Miner Upgrade Guide
    • CDN Nodes
      • Running a Cacher
      • Running a Retriever
    • TEE Nodes
      • What is TEE Node
      • Running a TEE Node
  • 🌏Community Members
    • Ambassador Program
    • Community Governance
  • 🔥CESS User Guide
    • CESS Account
    • Testnet Faucet
    • Territory Operation
  • 💻Developers
    • CESS SDK
      • Golang SDK
        • Preface
        • Install
        • Initialization
        • Properties
        • Data Process
        • Object/File
          • StoreFile
          • StoreObject
          • RetrieveFile
          • RetrieveObject
          • StoreFileToMiners
          • RetrieveFileFromMiners
        • Chain Related
          • Audit
            • QueryChallengeSnapShot
            • QueryCountedClear
            • QueryCountedServiceFailed
            • SubmitIdleProof
            • SubmitServiceProof
            • SubmitVerifyIdleResult
            • SubmitVerifyServiceResult
          • Babe
            • QueryAuthorities
          • Balances
            • QueryInactiveIssuance
            • QueryTotalIssuance
            • TransferToken
          • CessTreasury
            • QueryCurrencyReward
            • QueryEraReward
            • QueryReserveReward
            • QueryRoundReward
          • DeOSS
            • Authorize
            • CancelAuthorize
            • DestroyOss
            • QueryAllOss
            • QueryAuthorityList
            • QueryOss
            • RegisterOss
            • UpdateOss
          • FileBank
            • QueryAllRestoralOrder
            • QueryUserHoldFileList
            • QueryUserFidList
            • QueryDealMap
            • QueryFile
            • QueryRestoralOrder
            • CalculateReport
            • CertIdleSpace
            • ClaimRestoralNoExistOrder
            • ClaimRestoralOrder
            • DeleteFile
            • GenerateRestoralOrder
            • PlaceStorageOrder
            • ReplaceIdleSpace
            • RestoralOrderComplete
            • TransferReport
            • UploadDeclaration
            • TerritoryFileDelivery
          • SchedulerCredit
            • QueryCurrentCounters
          • Session
            • QueryValidators
          • Sminer
            • IncreaseCollateral
            • IncreaseDeclarationSpace
            • MinerExitPrep
            • MinerWithdraw
            • QueryAllMiner
            • QueryAllRestoralTarget
            • QueryCompleteSnapShot
            • QueryCounterForMinerItems
            • QueryExpenders
            • QueryMinerItems
            • QueryPendingReplacements
            • QueryRestoralTarget
            • QueryRewardMap
            • QueryStakingStartBlock
            • ReceiveReward
            • RegisterPoisKey
            • RegnstkAssignStaking
            • RegnstkSminer
            • UpdateBeneficiary
            • UpdateSminerEndPoint
          • Staking
            • QueryAllBonded
            • QueryAllNominators
            • QueryCounterForValidators
            • QueryCurrentEra
            • QueryEraValidatorReward
            • QueryErasRewardPoints
            • QueryErasTotalStake
            • QueryLedger
            • QueryNominatorCount
            • QueryValidatorCommission
            • QueryValidatorsCount
            • QueryeErasStakers
            • QueryeNominators
          • StorageHandler
            • QueryPurchasedSpace
            • QueryTotalIdleSpace
            • QueryTotalServiceSpace
            • QueryUnitPrice
            • QueryTerritory
            • QueryConsignment
            • MintTerritory
            • ExpandingTerritory
            • RenewalTerritory
            • ReactivateTerritory
            • TerritoryConsignment
            • CancelConsignment
            • BuyConsignment
            • CancelPurchaseAction
          • System
            • QueryAccountInfo
            • QueryAccountInfoByAccountID
            • QueryBlockNumber
          • Tee
            • QueryAllWorkers
            • QueryEndpoints
            • QueryMasterPubKey
            • QueryWorkerAddedAt
            • QueryWorkers
          • RPC Calls
            • ChainGetBlock
            • ChainGetBlockHash
            • ChainGetFinalizedHead
            • NetListening
            • SystemChain
            • SystemProperties
            • SystemSyncState
            • SystemVersion
        • Toolset
      • Javascript SDK
      • Rust SDK
        • Preface
        • chain
          • Audit
            • Challenge Snapshot
            • Counted Clear
            • Counted Service Failed
            • Submit Idle Space Proof
            • Submit Service Proof
            • Submit Verify Idle Result
            • Submit Verify Service Result
          • File Bank
            • Bucket
            • Clear User List
            • deal Map
            • File
            • Restoral Order
            • User Bucket List
            • User Hold File List
            • Calculate Report
            • Certify Idle Space
            • Claim Restoral Noexist Order
            • Claim Restoral Order
            • Create Bucket
            • Delete Bucket
            • Delete File
            • Generate Restoral Order
            • Replace Idle Space
            • Restoral Order Complete
            • Territory File Delivery
            • Transfer Report
            • Upload Declaration
          • Oss
            • Authority List
            • Oss
            • Authorize
            • Cancel Authorize
            • Destroy
            • Register
            • Update
          • Storage Handler
            • Consignment
            • Pay Order
            • Purchased Space
            • Territory Expired
            • Territory Frozen Counter
            • Territory Frozen
            • Territory
            • Territory Key
            • Total Power
            • Total Space
            • Unit Price
            • Buy Consignment
            • Cancel Consignment
            • Cancel Purchase Action
            • Create Order
            • Expand Territory
            • Mint Territory
            • Reactivate Territory
            • Renew Territory
            • Territory Consignment
            • Territory Grants
            • Territory Rename
    • Smart Contract
      • Issue ERC20
      • Deploy an ink! Smart Contract
      • Deploy a Solidity Smart Contract
      • Build a Proof of Existence dApp (ink!)
      • Build a NFT Marketplace (ink!)
      • Build a Proof of Existence dApp (Solidity)
    • Advanced Guides
      • Substrate and EVM Address Conversion
      • Commonly Used Libs
      • CESS Code Overview
  • 📱CESS Products
    • DeOSS
      • Introduction
      • Technical Highlights
      • Architecture
      • Data Access Process
      • API Description
        • Prerequisites
        • Identity Signature
        • Upload File/Object
        • Range Upload
        • Download File
        • Preview File
        • Delete File
        • View File Metadata
        • View Version
  • 📚References
    • 📚Node Operations
      • Running RPC Node
    • 📚In-depth Technical Features
      • R²S
      • RRC
      • LBSS
      • VRF
    • Data Lifecycle
      • Data Upload
      • Data Download
      • Data Deletion
      • Data Restore
      • Data Audit
    • Distributed Storage
      • Identification
      • Consistency Guarantee
      • Node Discovery
      • Message Protocol
      • Storage Method
    • DePIN
      • The DePIN Portrait
    • AI
      • CESS: The Future of Decentralized Data Infrastructure for the AI-Driven World
      • Decentralized Solutions for AI: How CESS is Shaping a Trusted Future
      • CESS in CES: Insights from Jensen Huang’s CES 2025 Speech and CESS Network’s Role in the AI Era
      • CESS AI Agent Hub: The Intelligent Portal of the AI Times
  • 📖Glossary
Powered by GitBook
On this page
  • Data Padding
  • Data Cutting
  • Data Redundancy
  • Merkle Hash

Was this helpful?

Edit on GitHub
  1. Developers
  2. CESS SDK
  3. Golang SDK

Data Process

Last updated 4 months ago

Was this helpful?

This section introduces the process of uploading files to the CESS network, how to ensure multiple copies of files, and how to recover files that have lost some data.

The data processing is divided into 3 steps in total: Step 1: Padding the data Step 2: Sequential cut of the data Step 3: Redundancy calculation is performed on the data Step 4: Calculate the Merkel hash of the data

Data Padding

Data padding is in order to unify the size of the data, but also for subsequent processing can be obtained in the form of uniform data specifications, at present, the CESS in accordance with the smallest integer multiples of 32MiB file padding, padding content is '0' (ASCII value is 48). Therefore, controlling the size of uploaded files to an integer multiple of 32MiB is the most effective use of space.

Example: For a 1MiB file: it willbe padded to 32MiB For 33MiB file: it will be padded to 64MiB For a 500MiB file: it will be padded to 512MiB And so on...

Please refer to the implementation:

Data Cutting

The data cut is a sequential cut of the data according to the 32MiB size, and since the data must be an integer multiple of 32MiB after padding, the number of chunks after the cut is also an integer. Due to the different sizes of the files, the final number of blocks after cutting is also not fixed. We call the cut block segment.

Please refer to the implementation:

Data Redundancy

Data redundancy is the redundancy calculation of the filled data, the redundant part of the data will be obtained, according to the amount of redundancy to determine the final size of the occupied space, CESS currently provides 2x the redundancy, plus 1x the original data, which is equivalent to 1 copy of the data occupies 3x space. The advantage of this multiple redundancy is that each redundant data is different, the redundant data and the body data are also different, so as to avoid exogenous attacks, this way allows the loss of any two data (body or redundancy can be) will be able to recover the original data.

CESS chooses 2x redundancy, 4 data slices and 8 redundancy slices, the slices are uniformly called fragment, see .

CESS redundancy algorithm adopts the , the specific implementation of the reference .

See for the implementation in sdk.

Merkle Hash

A file after padded, cut, redundant processing, will get a batch of segments and corresponding fragments, the number of segments based on the size of the file, and the number of fragments for each segment is a fixed number of 12, all the segments as a Merkle tree, the calculation of the root hash of the Merkle tree as the file's unique identifier, which we call fid.

Please refer to for the implementation of calculation merkle hash.

💻
FillAndCut
FillAndCut
pattern.go
Reed-Solomon
klauspost/reedsolomon
rs.go
hashtree.go